I have a WebApplication scoped feature, and I have set ActivateOnDefault to false, to prevent the feature from being activated on every web application when installed.
The feature is a simple HttpHandler, with FeatureActivated and FeatureDeactivating receivers to apply and remove web.config modifications.
When I activate the feature on one web app, both my web apps' web.config files have their modified by date updated to the current date, but only the one that the feature was activated on had the changes applied. (Central Admin is not affected).
I have basic logging in the event receivers and they are only running for the activated web app.