Is it possible to turn off certain constraints / annotations per class
at runtime? For instance if I wanted to turn of a @NotNull check on a
firstName field, is that possible?
No it is not. Bean Validation does not define such a feature. There is an open issue in Hibernate Validator HV-98 which discusses the possibility of reloading metadata, but even there you would need to rebuild the validator factory.
You could override annotations via XML configuration and then recreate the Validator(Factory) instance using different configurations at the time, but that's probably not easy to mange.
This would make testing to see whether a certain constraint is
triggered correctly simpler, as I could turn off all the other
constraints, and just check that one constraint.
If it is about testing, you can use Validator.validateValue to just validate a given field. Other than that, if you validate the whole object graph and get a set of constraint violations back, you can just iterate over them and inspect the metadata. There is enough information in the metadata to verify that a specific constraints was executed and failed.