Parallelism with Data from SqlDataReader

Question

I have a loop that processes data, row by row from a SqlDataReader. It works dandy. Now I want to add parallelism to it. Ideally I want to read a row, toss it to a thread, read another row, etc. I'd like a configurable number of threads (say 10) so as one opens up (completes its task), another is allowed to start.

Is there a built in way to do this, or should I handle it myself? I looked at PLINQ and Parallel, but I have trouble getting my head around it i think.

Here is a similar question asking the same thing: Is there a way to use the Task Parallel Library(TPL) with SQLDataReader?. Be sure to look at both the accepted answer and the other answer. — Scott Chamberlain
– Scott Chamberlain, Commented Jul 12, 2013 at 15:15

Mike Perrenoud · Accepted Answer · 2013-07-12 15:08:12Z

2

Since you're using a DataReader, it's forward only and one row at a time in memory, so you're going to need to handle this on your own. You're going to need to marshal the row in memory into some data structure and pass that on to the thread and then issue Read() again to move next.

You'll have to determine how you're going to pause when you have too many threads open already, but that's a whole other discussion and if you end up having questions about that then post what you have and we'll gladly help!

answered Jul 12, 2013 at 15:08

Mike Perrenoud

68.1k32 gold badges167 silver badges238 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Jonesopolis Over a year ago

Thanks Michael. In your opinion is there another/better option than DataReader? There's just too many rows to read it all into memory at once.

Jonesopolis Over a year ago

Also, do you have any idea where to find an example of implementing this? (Mostly the correct terminology for a google search)

Mike Perrenoud Over a year ago

@Jonesy, I don't think there really is a better way. I think you're doing it right. And further, there probably isn't anything really good to search for because this is a pretty custom implementation - I would just write it. In short, take the row in memory and put it into a new object and then queue that object for the background threads. I would recommending using a BackgroundWorker and the ConcurrentQueue, and I wouldn't worry about pausing unless you actually run into a problem. The ConcurrentQueue is thread safe so you can read/write from different threads safely.

Scott Chamberlain Over a year ago

The link I added to the original question has a example of some code on how to do it.

Collectives™ on Stack Overflow

Parallelism with Data from SqlDataReader

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related