I have written an application for my company that essentially sends records from a text or CSV file in arrays of 100 records to a web service, to which it then returns the response, also in arrays of 100 records and writes them to another file. Currently, it is single threaded (processing sequentially using the Backgroundworker in Windows Pro Forms), but I am looking to multi-thread it utilizing a threadpool and concurrentqueue.
#region NonDataCollectionService
if (!userService.needsAllRecords)
{
ConcurrentQueue<Record[]> outputQueue = new ConcurrentQueue<Record[]>();
while ((!inputFile.checkForEnd()) && (!backgroundWorker1.CancellationPending))
{
//Get array of (typically) 100 records from input file
Record[] serviceInputRecord = inputFile.getRecords(userService.maxRecsPerRequest);
//Queue array to be processed by threadpool, send in output concurrentqueue to be filled by threads
ThreadPool.QueueUserWorkItem(new WaitCallback(sendToService), new object[]{serviceInputRecord, outputQueue});
}
}
#endregion
void sendToService(Object stateInfo)
{
//The following block is in progress, I basically need to create copies of my class that calls the service for each thread
IWS threadService = Activator.CreateInstance(userService.GetType()) as IWS;
threadService.errorStatus = userService.errorStatus;
threadService.inputColumns = userService.inputColumns;
threadService.outputColumns = userService.outputColumns;
threadService.serviceOptions = userService.serviceOptions;
threadService.userLicense = userService.userLicense;
object[] objectArray = stateInfo as object[];
//Send input records to service
threadService.sendToService((Record[])objectArray[0]);
//The line below returns correctly
Record[] test123 = threadService.outputRecords;
ConcurrentQueue<Record[]> threadQueue = objectArray[1] as ConcurrentQueue<Record[]>;
//threadQueue has records here
threadQueue.Enqueue(test123);
}
However, when I check "outputQueue" in the top block, it is empty, even when threadQueue has records queued. Is this because I'm passing by value instead of passing by reference? If so, how would I do that syntactically with Threadpool.QueueUserWorkItem?
ThreadPoolthreads to complete. Also, you may be over-saturating theThreadPool, potentially. Might want to consider usingTask<T>here along withTask.WaitAll. By default aTaskwill also utilize theThreadPoolbut makes it much easier to manage concurrency.TransformBlock<TInput,TOutput>would probably serve you well here. Take a look at this question.