ASP.NET MVC Application crashing randomly

Question

I created a MVC 2 Application to work as a RSS Feeder and deliver News Contents for a number of applications to consume. Everything was doing fine until yesterday when suddenly my application started throwing a random error which doesn't tell me much of what's going on (or at least I don't understand it). This error only occurs in Production and cannot be reproduced in staging or my local machine.

Here's the stack trace:

Error executing child request for handler 'System.Web.Mvc.HttpHandlerUtil+ServerExecuteHttpHandlerWrapper'.errorPath:/Android/Edition/2011-11-22/P1 HostIP:##.##.##.## at System.Web.HttpServerUtility.ExecuteInternal(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage, VirtualPath path, VirtualPath filePath, String physPath, Exception error, String queryStringOverride) at System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter writer, Boolean preserveForm, Boolean setPreviousPage) at System.Web.HttpServerUtility.Execute(IHttpHandler handler, TextWriter writer, Boolean preserveForm) at System.Web.HttpServerUtilityWrapper.Execute(IHttpHandler handler, TextWriter writer, Boolean preserveForm) at System.Web.Mvc.ViewPage.RenderView(ViewContext viewContext) at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context) at System.Web.Mvc.ControllerActionInvoker.<>c_DisplayClass14.b_11() at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func1 continuation) at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func1 continuation) at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList1 filters, ActionResult actionResult) at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) at System.Web.Mvc.Controller.ExecuteCore() at System.Web.Mvc.MvcHandler.<>c__DisplayClass8.b__4() at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass1.b__0() at System.Web.Mvc.Async.AsyncResultWrapper.<>c__DisplayClass81.b__7(IAsyncResult _) at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.End() at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

This error only occurs sometimes and can throw at any action of any controller. I would like to have found the reason in a specific action but it only throws sometimes and can throw for any action.

/Android/Edition/2011-11-22/P1 is just a root path which leads to an action called Edition. I displayed the stack trace when the error triggered in this action but it does trigger in all other actions as well. — Hallaghan
– Hallaghan, Commented Nov 30, 2011 at 12:39
MMmmm okay, not relevant then. Maybe this link can help: stackoverflow.com/questions/4862008/… — Francisco
– Francisco, Commented Nov 30, 2011 at 12:44

mnemosyn · Accepted Answer · 2011-11-30 14:49:59Z

2

Such byzantine errors are hard to debug. The stack trace you're seeing does not help in diagnosing the error.

First, you should improve error logging. In your global.asax, create an implementation of the Application_Error hook that logs the exception and all inner exceptions to a file (I wouldn't log this to the database, because the db connection could be the culprit). Make sure that this code is solid: it should focus on these critical errors and not log every 404 page. Also, make sure that the logging code itself does not create any problems (it should be highly error tolerant).

The cause for this kind of problems is usually access to some kind of static variables. Out of all keywords, I believe static is by far the most dangerous one because it is so subtle.

Some common errors I've seen.

Caching Someone wanted to be smart and cache some data in a static dictionary or so. Unfortunately, the locking code is flawed. An exception occurs only if the code for some user tries to add sth. to the cache, but it's already there:
```
if(_dict.ContainsKey(cacheKey) == false)
{
   // second thread adds data to the dictionary here
   _dict.Add(cacheKey, cacheData); // exception
}
```
This can also happen in a 3rd party library that uses caching or pooling. Access static variables with care.
Uncommon Code Paths Something unusual happens that calls code that is not called very often, and that code is flawed. If you have testing with high code coverage, check the spots that are not covered by the tests.
DB Connection Lost A socket reset on the db connection could lead to exceptions. Many database connection libraries / drivers fix this quickly (i.e. the next request will be fine). This is hard to handle; basically it shouldn't happen but there are many reasons why it could occur.

edited Nov 30, 2011 at 14:49

answered Nov 30, 2011 at 14:38

mnemosyn

46.4k6 gold badges80 silver badges84 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Hallaghan Over a year ago

Your answer is quite informative and I will be investigating the matters you mentioned. Quite frankly, I do some caching for some actions in this application but that caching is not used for the opening actions of the application. Is it possible that when the app for some reason crashes will take a while to restart working normally? And to add up, I'm using an implementation of Application_Error in global.asax but I will improve it accordingly with your tips. Feedback later today.

mnemosyn Over a year ago

Hehe, caching is fine of course -- but it's easy to add subtle bugs :) There's a big delay if the entire app has to restart, because the views are usually compiled on-the-fly. If it's a caching problem, maybe your cache is empty an now all data must be read again. Good luck.

Hallaghan Over a year ago

I somehow got it to reproduce in local environment by spamming F5 on this page where the exception was throwing. I got it still in debug and on hold while I'm examining the variables. Do you got any tips that I could use to help investigate this problem? Also do you think that the fact I spammed refresh on the action could be part of this problem?

mnemosyn Over a year ago

Yes. In a production environment, there are more requests, so there is a high chance that two threads will access a static variable at the same time. That does not usually happen in a single-session debug. If you keep hitting the local machine, you'll spawn up multiple threads as well. Using the debugger is tricky, because it's now multi-threaded so you can't step through your code. I'd suggest you enable the 'break on first-chance exception' option and try to find the error that way.

Hallaghan Over a year ago

I will try to enable it and debug that way. Is it possible this exception is triggered by a request overflow to this action? There are no static variables being used/called in this particular action. I will get back at you once I find more.

|

rouen · Accepted Answer · 2011-11-30 11:50:30Z

1

This error tells you, that the true error is somewhere in some child action, that is action rendered by Html.RenderAction().

There is no such thing as "random" error - there has to be scenario where it happens, you just dont know it because the conditions vary maybe.

You should identify all your child actions, and test them with various inputs.. after you identify the problematic input set, try disabling child actions one by one and find the error this way.

answered Nov 30, 2011 at 11:50

rouen

5,1742 gold badges28 silver badges52 bronze badges

4 Comments

Hallaghan Over a year ago

random is a way of naming it when I ain't sure of what's throwing this error which, like I said, does happen for every Action of any Controller in my app. Sometimes it can spend hours without happening. A detail worth mentioning is that there's a page which I load with data from the database and the Action responsible receives no parameter. So it loads always the same way. This action throws this exception sometimes and sometimes it doesn't. To be true, it really rarely happens.

rouen Over a year ago

Again - the error is in some child action, called by Html.RenderAction() from your view or maybe even layout. This way it can seems that "every action can crash", but that is not true - your child action "included" in result of all actions can crash. Examine your child actions.

Hallaghan Over a year ago

Rouen, I don't really have child actions, and I didn't even know they existed for MVC2. Any ideas?

rouen Over a year ago

I am pretty sure child action is behind this.. if it is not in your case, then i have no idea...

Collectives™ on Stack Overflow

ASP.NET MVC Application crashing randomly

2 Answers 2

8 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

8 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related