1

I have an application, which need to do some validation on the database end. Each process goes through database validation (search on the table that has around 6 million records), then goes through some Lucene Index searcher (the index is built off this 6 million record table). As these steps are disjoint for each line items being passed, I am thinking of utilizing multicore threading. (each of these lines take around 1/2 minute on a single thread).

What are my options with multicore in C#? Is there some good resources / third party library (I looked a bit at PowerThreading by Jeff Ritcher), some good tutorials.

I assume I need to do some thread pools in N core machines.

Currently, it takes around 40 secs to process 100 lines, looking to get this done to around 10 secs.

Thanks...

Improve this question
2
  • Is this just a duplicate? It sounds like it amounts to 'how do I write multithreaded code in .NET?' Searching on "multithreading [c#]" turns up a lot of useful information. For example, see the accepted answer to stackoverflow.com/questions/100291/… for a sample technique. Commented Sep 23, 2009 at 18:53
  • I somehow feel that SQL part is not optimized yet and it is early to start with app multithreading. Commented Sep 30, 2009 at 11:44

3 Answers 3

Reset to default
1

Simple threading should give you access to the multicore. You will have to play around with the size of the thread pool, as your tasks look to have a lot of IO as well.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

when you mean simple threading you are talking about threading pool? I tried threading pool in double core machine and still get around same performance!
Remember what Chris said about decoupling interdependencies?
I admit I haven't tried checking whether multi-threads use multiple cores, but that is true for Java and specifically noted as not true for Python because of a global lock. Also synchronization and Amdahl's law (en.wikipedia.org/wiki/Amdahl%27s_law) will govern in any case.
Yes, multiple threads will be spread across all of your cores unless you explicitly prevent this. If the threads work independently then you should get a speed-up nearly proportional to the number of cores. If you're not getting that, something is wrong, most likely a global lock.
If the computation is independent and IO bound, you may be able to get greater speedup than the number of cores.
1

Have you looked into F#?

It is designed from the ground up for parallelizing tasks.

Improve this answer

2 Comments

I dont' have enough bandwith to use F# right now. I will look into it. But, is there some C# solution you can suggest?
Keep your threads decoupled from interdependencies as much as possible.
0

If you want a factor of four speed increase and have four cores, all you need to do is avoid dependencies among tasks, which sounds feasible. I think you'll find that you'll actually want to run more threads than cores because, at any given time, a number of threads will be blocked waiting for I/O. Therefore, I suggest that, whichever method you use, ensure that it's easy to benchmark it with different thread counts.

Improve this answer

1 Comment

Well, the simplest thing is to use the ThreadPool, as per msdn.microsoft.com/en-us/library/ms973903.aspx.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.