Yes, that is the way it works, and the way it should work.
The reason is that attributes do not magically work. Something must examine them and do things based on them. That code is part of the normal pipeline of MVC when processing a request.
When you call that method directly, you bypass the normal pipeline, thus attributes do not get evaluated and/or run.
Beyond that, just simple common sense should tell you that caching only applies to a response. If there is no response (because you called it directly) it would not apply.
What you should do is execute a child action instead. How you would do that differs depending on where and how you are calling the method. So unless you show us the code that calls the method (and its surrounding context) i can't help you further.