How to return 401 for REST API call in Spring Security?

I came to more simpler solution. In Spring Boot and Java config you just have to register additional entry point in addition to default one. And because all your rest-services resides in "/api" name space, you could use AntPathRequestMatcher("/api/**") to match necessary requests.

So, final solution is:

@Configuration
@EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
    http.exceptionHandling()
                    //Actually Spring already configures default AuthenticationEntryPoint - LoginUrlAuthenticationEntryPoint
                    //This one is REST-specific addition to default one, that is based on PathRequest
                    .defaultAuthenticationEntryPointFor(getRestAuthenticationEntryPoint(), new AntPathRequestMatcher("/api/**"));
    }

    private AuthenticationEntryPoint getRestAuthenticationEntryPoint() {
        return new HttpStatusEntryPoint(HttpStatus.UNAUTHORIZED);
    }
}

You need to have a custom authentication entry point.

public class CustomEntryPoint extends LoginUrlAuthenticationEntryPoint {

    private static final String XML_HTTP_REQUEST = "XMLHttpRequest";
    private static final String X_REQUESTED_WITH = "X-Requested-With";

    public CustomEntryPoint(String loginFormUrl) {
        super(loginFormUrl);
    }

    @Override
    public void commence(HttpServletRequest request, HttpServletResponse response, AuthenticationException exception)
            throws IOException, ServletException {
        if (XML_HTTP_REQUEST.equals(request.getHeader(X_REQUESTED_WITH))) {
            response.sendError(HttpServletResponse.SC_UNAUTHORIZED);
        } else {
            super.commence(request, response, exception);
        }
    }    
}

Finally change your config to this:

<security:http auto-config='true' use-expressions="true" entry-point-ref="customEntryPoint">
    <security:intercept-url pattern="/api*" access="hasRole('USER')" />
    <security:intercept-url pattern="/oldweb*" access="hasRole('USER')" />

    <security:form-login login-page="/login.jsp" authentication-failure-url="/login.jsp?login_error=1" default-target-url="/home"/>

    <beans:bean id="customEntryPoint" class="CustomEntryPoint">
        <beans:constructor-arg value="/login"/>
    </beans:bean>    
</security:http>

When text/html is not acceptable, return 401.

@EnableWebSecurity
public class OAuth2LoginSecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(final HttpSecurity http) throws Exception {
        // ...
        final RequestMatcher matcher = new NegatedRequestMatcher(
            new MediaTypeRequestMatcher(MediaType.TEXT_HTML));

        http.exceptionHandling()
            .defaultAuthenticationEntryPointFor(
                new HttpStatusEntryPoint(HttpStatus.UNAUTHORIZED), matcher);

    }
}