20

I have a method that calls a SQLServer function to perform a free text search against a table. That function will occasionally on the first call result in a SQLException: "Word breaking timed out for the full-text query string". So typically I want to retry that request because it will succeed on subsequent requests. What is good style for structuring the retry logic. At the moment I have the following:

var retryCount = 0;
var results = new List<UserSummaryDto>();
using (var ctx = new UsersDataContext(ConfigurationManager.ConnectionStrings[CONNECTION_STRING_KEY].ConnectionString))
{
    for (; ; )
    {
        try
        {
            results = ctx.SearchPhoneList(value, maxRows)
                         .Select(user => user.ToDto())
                         .ToList();
            break;
        }
        catch (SqlException)
        {
            retryCount++;
            if (retryCount > MAX_RETRY) throw;
        }
    }
}

return results;
5
  • 1
    Actually, its conceptually simple and should work. Although I'd caution against the exteme use of Var. Sometimes an int is just an int. Commented Jan 27, 2011 at 22:00
  • On Error Resume Next in Visual Basic was conceptually simple - that didn't make it a good idea. Commented Jan 28, 2011 at 0:12
  • @MAW74656: I voted your comment up on the basis of the var remark. IMO, the compiler should disallow its use where it conceals the object type, as it does here. Commented Jan 28, 2011 at 0:14
  • 5
    Just habit, I like to explicitly initialise my variables. Plus I use ReSharper which will suggest replacing declared types with "var" and tell me that "int i = 0;" doesn't need to be initialised to zero because it is by default. Cautioning against the extreme use of var may be a little extreme. Commented Jan 28, 2011 at 0:46
  • 1
    Just an update on the var comments. There isn't anything in the above code that conceals the object type. retryCount is clearly int, results is a generic List, ctx is a data context. I did recently have a situation where a variable was initialised by a method where the returned object was not obvious e.g. var someVar = MyInitialiser();. Under those circumstances a var declaration conceals the type of the variable and should be declared explicitly. Commented May 6, 2015 at 20:51

8 Answers 8

17

I'd change the exception handling to only retry on certain errors:

  • 1204, 1205 deadlocks
  • -2 timeout
  • -1 connection broken

These are the basic "retryable" errors

catch (SqlException ex)
{
    if !(ex.Number == 1205 || ex.Number == 1204 || ... )
    {
        throw
    }
    retryCount++;
    if (retryCount > MAX_RETRY) throw;
}

Edit, I clean forgot about waits so you don't hammer the SQL box:

  • Add a 500 ms wait on deadlock
  • Add a 5 sec delay on timeout

Edit 2:

I'm a Developer DBA, don't do much C#. My answer was to correct exception processing for the calls...

Sign up to request clarification or add additional context in comments.

8 Comments

@jani: Your link may improve the quality of c# (I'm a developer DBA: c# isn't a core skill for me) but it doesn't help with trapping database errors nicely to work out if a retry is possible.
This is OK for the purpose. The one thing I'd do is group the error codes in an array and use .Contains(ex.Number); makes it a little more compact.
@Keith and what if your message contains two of them? relying on the error message returned by db like this, is not a good idea IMHO.
I think retrying in your store procedure would be much more better than retry in application level, it's more fast, less error prone, and doesn't need a round trip between the App and DB.
@Jani - What? An exception will be for a single error, usually the first one encountered by the engine when processing a command. And we're not relying on the message, but on the number, which is unique to a type of error and the ONLY way you can tell one AdoException or SqlException from another.
|
12

Thanks for all the feedback. I'm answering this myself so I can incorporate elements from the answers given. Please let me know if I've missed something. My method becomes:

var results = new List<UserSummaryDto>();
Retry<UsersDataContext>(ctx => results = ctx.SearchPhoneList(value, maxRows)
                                            .Select(user => user.ToDto())
                                            .ToList());
return results;

And I've refactored the original method for reuse. Still lots of levels of nesting. It also relies on there being a default constructor for the data context which may be too restrictive. @Martin, I considered including your PreserveStackTrace method but in this case I don't think it really adds enough value - good to know for future reference thanks:

private const int MAX_RETRY = 2;
private const double LONG_WAIT_SECONDS = 5;
private const double SHORT_WAIT_SECONDS = 0.5;
private static readonly TimeSpan longWait = TimeSpan.FromSeconds(LONG_WAIT_SECONDS);
private static readonly TimeSpan shortWait = TimeSpan.FromSeconds(SHORT_WAIT_SECONDS);
private enum RetryableSqlErrors
{
    Timeout = -2,
    NoLock = 1204,
    Deadlock = 1205,
    WordbreakerTimeout = 30053,
}

private void Retry<T>(Action<T> retryAction) where T : DataContext, new()
{
    var retryCount = 0;
    using (var ctx = new T())
    {
        for (;;)
        {
            try
            {
                retryAction(ctx);
                break;
            }
            catch (SqlException ex)
                when (ex.Number == (int) RetryableSqlErrors.Timeout &&
                      retryCount < MAX_RETRY)
            {
                Thread.Sleep(longWait);
            }
            catch (SqlException ex)
                when (Enum.IsDefined(typeof(RetryableSqlErrors), ex.Number) &&
                      retryCount < MAX_RETRY)
            {
                Thread.Sleep(shortWait);
            }
            retryCount++;
        }
    }
}

1 Comment

8

My enum of retryables for sql looks like this:

SqlConnectionBroken = -1,
SqlTimeout = -2,
SqlOutOfMemory = 701,
SqlOutOfLocks = 1204,
SqlDeadlockVictim = 1205,
SqlLockRequestTimeout = 1222,
SqlTimeoutWaitingForMemoryResource = 8645,
SqlLowMemoryCondition = 8651,
SqlWordbreakerTimeout = 30053

1 Comment

7

It's not good style, but sometimes you have to do it, because you simply can't change existing code and have to deal with it.

I am using the following generic method for this scenario. Note the PreserveStackTrace() method, which can sometimes be very helpful in a re-throw scenario.

public static void RetryBeforeThrow<T>(Action action, int retries, int timeout) where T : Exception
{
    if (action == null)
        throw new ArgumentNullException("action", string.Format("Argument '{0}' cannot be null.", "action"));

    int tries = 1;

    do
    {
        try
        {
            action();
            return;
        }
        catch (T ex)
        {
            if (retries <= 0)
            {
                PreserveStackTrace(ex);
                throw;
            }

            Thread.Sleep(timeout);
        }
    }
    while (tries++ < retries);
}

/// <summary>
/// Sets a flag on an <see cref="T:System.Exception"/> so that all the stack trace information is preserved 
/// when the exception is re-thrown.
/// </summary>
/// <remarks>This is useful because "throw" removes information, such as the original stack frame.</remarks>
/// <see href="http://weblogs.asp.net/fmarguerie/archive/2008/01/02/rethrowing-exceptions-and-preserving-the-full-call-stack-trace.aspx"/>
public static void PreserveStackTrace(Exception ex)
{
    MethodInfo preserveStackTrace = typeof(Exception).GetMethod("InternalPreserveStackTrace", BindingFlags.Instance | BindingFlags.NonPublic);
    preserveStackTrace.Invoke(ex, null);
}

You would call it like that:

RetryBeforeThrow<SqlException>(() => MethodWhichFails(), 3, 100);

4 Comments

throw should not remove original stack frame. Only if you used throw ex
@František Žiačik Generaly it should. Read the article on Fabrice blog in my answer and you'll see what I mean.
I tried it myself and it turns out you're right. One never stops learning. The other way to preserve the information is to wrap the exception (throw new Exception("Message", ex))
Better solution is ExceptionDispatchInfo view the answer , not use PreserveStackTrace ? anyways, better use PrepForRemoting if not ExceptionDispatchInfo available
3

There is no good style for doing something like this. You'd be better off figuring out why the request fails the first time but succeeds the second time.

It seems possible that Sql Server has to initially compile an execution plan and then execute the query. So the first call fails because the combined times exceed your timeout property, and succeeds the second time because the execution plan is already compiled and saved.

I don't know how UsersDataContext works, but it may be the case that you have the option to Prepare the query before actually executing it.

Real Answer: If I had to do this, I would retry just once and not again, like this:

var results = new List<UserSummaryDto>();
using (var ctx = new 
    UsersDataContext(ConfigurationManager.ConnectionStrings[CONNECTION_STRING_KEY].ConnectionString))
{
        try
        {
            results = ctx.SearchPhoneList(value, maxRows)
                         .Select(user => user.ToDto())
                         .ToList();
            break;
        }
        catch (SqlException)
        {
            try
            {
                results = ctx.SearchPhoneList(value, maxRows)
                         .Select(user => user.ToDto())
                         .ToList();
                break;
            }
            catch (SqlException)
            {
                // set return value, or indicate failure to user however
            }
        }
    }
}

return results;

While I might trust you to not abuse the retry process, you'd be tempting your successor to increase the retry count as a quick fix.

16 Comments

I was going to say this, but was too chicken. If timeouts are a big problem, and getting the system to where it can stand up without having to do retries, I would consider smoothing out the requests into a queue and then responding asynchronously.
I disagree: some errors in SQL server are inherently retryable. If you hit the server during index rebuild for example, a wait and retry is harmless and avoids unnecessary support or logging
@gbn and David Clarke: if you guys are comfortable being responsible for an app that fails like this, then more power to you. I wouldn't be at all comfortable with it, and would put it at the top of my "must fix this" list.
@MusiGenesis if you've ever been involved with enterprise-level development you might take a more defensive approach to coding. Yes it is good to identify why something is failing but it doesn't help the poor user whose app starts crashing because someone introduced a new stored proc into the database that locks the requested rows. If you allow a retry then there is at least a possibility for the user to continue their work, albeit more slowly.
@MusiGenesis clearly we disagree. To me this is neither a desperate emergency fix or a solution, it is a defensive measure to allow for some flexibility when dealing with an enterprise system that isn't under my sole control. There are a variety of issues that can occur on a production database that shouldn't result in the user's application failing. With respect to your last comment, you'll be concerned to know that I've worked previously for a number of banks - perhaps you should consider keeping your money under your mattress ;-)
|
1

I think annotating a method with an aspect specifying the retry count would result in more structured code, although it needs some infrastructure coding.

Comments

0

You can simply use SqlConnectionStringBuilder properties to sql connection retry.

var conBuilder = new SqlConnectionStringBuilder("Server=.;Database=xxxx;Trusted_Connection=True;MultipleActiveResultSets=true"); conBuilder.ConnectTimeout = 90; conBuilder.ConnectRetryInterval = 15; conBuilder.ConnectRetryCount = 6;

Note:- Required .Net 4.5 or later.

Comments

-7

Pull the relevant code out into its own method, then use recursion.

Pseudo-code:

try
{
    doDatabaseCall();
}
catch (exception e)
{
    //Check exception object to confirm its the error you've been experiencing as opposed to the server being offline.
    doDatabaseCall();
}

6 Comments

And if the server is offline for 2 hours you propose to keep trying...?
@gbn- No, of course not. We're only giving him concepts here, not writing his entire code. See my edit.
Maybe some combination of his counter and my abstraction would be best.
I don't see any recursion in your code. Anyway, it doesn't seem to me like a good idea to use recursion for this.
I marked this as low quality, and a moderator declined my flag. SO is broken.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.