Is there maybe a more stripped down starting point I should use if the VS template is adding a lot of stuff I don't need?
These VS templates are good to trim down initial setup and learning curve in the beginning, however you would need to customize them for your real project needs.
For example, on this site http://www.reviewstoshare.com, my friend using AngularJS along with ASP.NET MVC. Keep in mind that this site was already built using ASP.MVC + Jquery for in page interaction as needed.
On the other hand there is still some "Ajaxy" nature to the site like comments, voting, flagging etc. Not too different than Stackoverflow itself. Before AngularJS it was a mess of Jquery plugins and functions within $(document).ready() callback, not to mention the JS code was not testable much.
Overall, having a proper mix of bothmix of both make your site veryslick and functional.

Some good references to look: