Spring Boot @ExceptionHandler working on dev server but not locally

I know this is usually an issue that happens the other way around, so I am caught a little of guard here :D

I have built a user-management backend that provides a UI with data. When this architecture is deployed on our dev-server, everything works beautifully. However, as soon as I try to run the integration tests (which we do using a maven cargo tomcat) or if I use the war file in a local tomcat, the exception handlers aren't used at all. Spring simply displays a standard 500 response with the exception transformed into the body.

Perusing SO for similar issues has only resulted in the advice that I should use @EnableWebMVC, but that is neither applicable to what my backend is trying to accomplish, nor does it change anything.

How should I go about looking for the solution to this issue? Specifically, can I somehow observe if my controlleradvice is even scanned, and is there a reason why it might not be?

EDIT: These are the relevant files:

SpringConfiguration:

@Configuration
@ComponentScan(basePackageClasses = {UserManagementSpringConfiguration.class})
@EnableWebSecurity
public class UserManagementSpringConfiguration {

@Configuration
public static class ResourceMappingConfig implements WebMvcConfigurer {

    @Override
    public void addViewControllers(ViewControllerRegistry registry) {
        registry.addViewController("/ui/*/usermanagement").setViewName("forward:/usermanagement-ui/index.html");
//            registry.addViewController("/ui/*/*/generator/").setViewName("forward:/generator-ui/index.html");
        registry.addViewController("/ui/*/usermanagement/*").setViewName("forward:/usermanagement-ui/index.html");
    }

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
        // cache setting, otherwise fonts are not loaded in IE over https
        CacheControl cacheControl = CacheControl.noCache().mustRevalidate();
        registry.addResourceHandler("/ui/**/*").addResourceLocations("/usermanagement-ui/")
                    .setCacheControl(cacheControl);
        }

    }

}

ControllerAdvice:

@ControllerAdvice
public class CustomResponseEntityExceptionHandler {

    public static final Logger LOG = EISLoggerFactory.getLogger(CustomResponseEntityExceptionHandler.class);

    @PostConstruct
    public void postConstruct() {
        LOG.debug("CustomExceptionHandler loaded and ready for use");
    }

    @ExceptionHandler(PasswordMismatchException.class)
    public final ResponseEntity<ErrorDetails> handlePasswordChangeMismatch(
            PasswordMismatchException ex,
            WebRequest request) {
        ErrorDetails errorDetails = new ErrorDetails(
                new Date(),
                ex.getMessage(),
                request.getDescription(false),
                MessageKeys.mismatchedPassword);
        return new ResponseEntity<>(errorDetails, HttpStatus.BAD_REQUEST);
    }
}