Running Multiple Tasks in Parallel

Question

I have a list of proxies, each proxy goes to various sites and pulls the needed data from the sites. Currently it's doing this one at a time. But I'd like to have 10 - 20 tasks running at once so it's downloading from 20 sites in one go rather than just one.

Here's how I'm currently doing it:

private async Task<string> DownloadDataFromSite(string url)
{
     // (await) Do Work.
    return HTMLSourceCode;
}

I then loop through the proxies

foreach(Proxy p in proxies)
{
    string source = await DownloadDataFromSite(site);
}

Is Parallel.ForEach suitable for such a task? I've tried it, but the problem I'm having at the moment is not being able to await.

Luaan · Accepted Answer · 2014-05-19 12:36:38Z

5

One way is to avoid awaiting in the foreach. The thing is that your await effectively blocks your execution. A better way might be something like this:

await Task.WhenAll(proxies.Select(p => DownloadDataFromSite(site)));

This will mean you'll be awaiting all the tasks at once, which means the asynchronous I/O is going to happen in parallel. Note that if you're doing CPU work too, that's not going to really be parallelized.

The point is, asynchronous I/O (such as downloading a web page) doesn't require more threads to run in parallel. On the other hand, Parallel.ForEach is actually intended for CPU-bound work, rather than I/O bound work, and it does execute the code on multiple threads.

answered May 19, 2014 at 12:36

Luaan

64.1k7 gold badges108 silver badges126 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

James Jeffery Over a year ago

Interesting. Is Select intended to be used in this way? I assumed that Select is used to apply some type of transform method.

Luaan Over a year ago

@JamesJeffery Well, that is a transformation method - it transforms an input parameter (site) into a Task :P

Community · Accepted Answer · 2017-05-23 11:50:23Z

0

PArallel.ForEach does not work well as it expects a synchronous lambda and giving it an asynchronous one basically causes it to return as soon as it starts. There is a way around it though, check this question out: Is it OK to do some async/await inside some .NET Parallel.ForEach() code?

edited May 23, 2017 at 11:50

CommunityBot

11 silver badge

answered May 19, 2014 at 12:36

NeddySpaghetti

13.6k5 gold badges35 silver badges63 bronze badges

Collectives™ on Stack Overflow

Running Multiple Tasks in Parallel

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related