ASP.NET Web app with large amount of threads, most efficient method

Question

I have a web app that allows you to paste a large number of urls in (anywhere from 2000 - 15000), it then visits each page to download the source and do some processing.

I've tried a couple methods, currently it uses ajax to send each url one at a time to a WebMethod which then uses ThreadPool.QueueUserWorkItem to push the request onto a queue and do the processing when it can so that many threads are running at the same time. This seems to cause problem (the Visual Studio web server dies occasionally which can't be good).

Just wondering if anyones done anything similar, knows a better way. I guess being able to throttle the number of concurrent threads somehow would help.

Thanks

Cassini is not meant to handle this kind of load. You should use it for what it is for. — O.O
– O.O, Commented Jan 13, 2012 at 21:08

Jordan · Accepted Answer · 2012-01-13 21:05:26Z

2

Don't use the built in webserver. Use IIS proper

answered Jan 13, 2012 at 21:05

Jordan

2,7584 gold badges24 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

JeremyBeadle Over a year ago

I was going to change it (still will), but figured it's probably more to do with the amount of work it is doing in one go rather than the webserver?

Jordan Over a year ago

My guess is it is the webserver .... check this out developer.com/net/… It is relatively easy to try setting it up with IIS7.5 Express and try it. Also what kind of processing is it doing? If you are doing multi-threading stuff perhaps you are running into deadlocks and such, not sure how much you know on that topic or precisely what you are doing so here's a link msdn.microsoft.com/en-us/magazine/cc163618.aspx

Community · Accepted Answer · 2017-05-23 12:12:10Z

1

Along with using IIS as mentioned above, you might benefit from using asynchronous web requests. (See How to use HttpWebRequest (.NET) asynchronously? for more info.)

This will prevent the amount of blocking done on each thread, which should reduce the number of threads needed if your processing isn't too heavy.

edited May 23, 2017 at 12:12

CommunityBot

11 silver badge

answered Jan 14, 2012 at 18:50

Alex Moore

3,4751 gold badge26 silver badges39 bronze badges

Collectives™ on Stack Overflow

ASP.NET Web app with large amount of threads, most efficient method

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related