How to use Task.WhenAny and implement retry

Question

I have a solution that creates multiple I/O based tasks and I'm using Task.WhenAny() to manage these tasks. But often many of the tasks will fail due to network issue or request throttling etc.

I can't seem to find a solution that enables me to successfully retry failed tasks when using a Task.WhenAny() approach.

Here is what I'm doing:

var tasks = new List<Task<MyType>>();
foreach(var item in someCollection)
{
   task.Add(GetSomethingAsync());
}
while (tasks.Count > 0)
{
   var child = await Task.WhenAny(tasks);
   tasks.Remove(child);
   ???
}

So the above structure works for completing tasks, but I haven't found a way to handle and retry failing tasks. The await Task.WhenAny throws an AggregateException rather than allowing me to inspect a task status. When In the exception handler I no longer have any way to retry the failed task.

Can you please clarify what the difference between your tasks? Currently it looks like you have list of equal tasks. Maybe you are passing to GetSomethingAsync() some parameters which are different? — Sergey Berezovskiy
– Sergey Berezovskiy, Commented May 3, 2017 at 15:47
@Sergey, yes I'm passing a unique parameter to GetSomethingAsync which I'd need to be able to reference when retrying s task. — Matt Cotton
– Matt Cotton, Commented May 4, 2017 at 9:20

Stephen Cleary · Accepted Answer · 2017-05-03 15:47:25Z

9

I believe it would be easier to retry within the tasks, and then replace the Task.WhenAny-in-a-loop antipattern with Task.WhenAll

E.g., using Polly:

var tasks = new List<Task<MyType>>();
var policy = ...; // See Polly documentation
foreach(var item in someCollection)
  tasks.Add(policy.ExecuteAsync(() => GetSomethingAsync()));
await Task.WhenAll(tasks);

or, more succinctly:

var policy = ...; // See Polly documentation
var tasks = someCollection.Select(item => policy.ExecuteAsync(() => GetSomethingAsync()));
await Task.WhenAll(tasks);

answered May 3, 2017 at 15:47

Stephen Cleary

460k78 gold badges721 silver badges858 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Matt Cotton Over a year ago

thanks @Stephen, I'll try it out. I had dabbled with Polly for this but was still using the await Task.WhenAny pattern which didn't lead anywhere.

Stephen Cleary Over a year ago

@macon: No, it's just awkward. There's always a cleaner solution than having Task.WhenAny removing tasks as they complete (eventually waiting for them all to complete).

Theodor Zoulias · Accepted Answer · 2023-07-21 09:57:06Z

If you don't want to use the Polly library for some reason, you could use the Retry method below. It accepts a task factory, and keeps creating and then awaiting a task until it completes successfully, or the maxAttempts have been reached:

public static async Task<TResult> Retry<TResult>(Func<Task<TResult>> taskFactory,
    int maxAttempts)
{
    int failedAttempts = 0;
    while (true)
    {
        try
        {
            Task<TResult> task = taskFactory();
            return await task.ConfigureAwait(false);
        }
        catch
        {
            failedAttempts++;
            if (failedAttempts >= maxAttempts) throw;
        }
    }
}

You could then use this method to download (for example) some web pages.

string[] urls =
{
    "https://stackoverflow.com",
    "https://superuser.com",
    //"https://no-such.url",
};
var httpClient = new HttpClient();
var tasks = urls.Select(url => Retry(async () =>
{
    return (Url: url, Html: await httpClient.GetStringAsync(url));
}, maxAttempts: 5));

var results = await Task.WhenAll(tasks);
foreach (var result in results)
{
    Console.WriteLine($"Url: {result.Url}, {result.Html.Length:#,0} chars");
}

Output:

Url: https://stackoverflow.com, 112,276 chars  
Url: https://superuser.com, 122,784 chars

If you uncomment the third url then instead of these results an HttpRequestException will be thrown, after five failed attempts.

The Task.WhenAll method will wait for the completion of all tasks before propagating the error. In case it is preferable to report the error as soon as possible, you can find solutions in this question.

Collectives™ on Stack Overflow

How to use Task.WhenAny and implement retry

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related