Hashing on multiple keys : for task execution In Multi threaded environment

Question

I have certain objects on which certain tasks needs to be performed.On all objects all task needs to be performed. I want to employ multiple threads say N parallel threads

Say I have objects identifiers like A,B,C (Objects can be in 100 K range ; keys can be long or string) And Tasks can T1,T2,T3,TN - (Task are max 20 in number)

Conditions for task execution - Tasks can be executed in parallel even for the same object. But for the same object, for a given task, it should be executed in series. Example , say I have Objects on which are task performed are A,B,A and tasks are t1, t2

So T1(A), T2(A) or T1(A) , T2(B) are possible , but T1(A) and T1(A) shouldnt be allowed

How can I ensure that , that my conditions are met. I know I have to use some sort of hashing. I read about hashing , so my hash function can be of -

return ObjectIdentifier.getHashCode() + TaskIdentifier.getHashCode() or other can be - a^3 + b^2 (where a and b are hashes of object identifier and task identifier respectively) What would be best strategy, any suggestions

My task doesnt involve any IO, and as of now I am using one thread for each task. So my current design is ok, or should I try to optimize it based on num of processors. (have fixed num of threads )

Thanks for your answer, Hello, My my ListOfObjects are series of message which I am recv from external program (via a blockingcollection). And after completing n number of messages, or recv a condition, I have to stop processing the message, take care of that condition, and then resume the processing (continue with the blocking collection loop). I am unable understand how can I implement the above with messages coming from blocking collection — Rajesh Gaur
– Rajesh Gaur, Commented Sep 26, 2014 at 0:47

littlerufe · Accepted Answer · 2014-09-25 00:13:37Z

1

You can do a Parallel.ForEach on one of the lists, and a regular foreach on the other list, for example:

Parallel.ForEach (myListOfObjects, currentObject =>
{
    foreach(var task in myListOfTasks)
    {
        task.DoSomething(currentObject);
    }
});

answered Sep 25, 2014 at 0:13

littlerufe

37.2k5 gold badges33 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Kirill Shlenskiy · Accepted Answer · 2014-09-25 11:53:23Z

I must say that I really like Rufus L's answer. You have to be smart about the things you parallelise and not over-encumber your implementation with excessive thread synchronisation and memory-intensive constructs - those things diminish the benefit of parallelisation. Given the large size of the item pool and the CPU-bound nature of the work, Parallel.ForEach with a sequential inner loop should provide very reasonable performance while keeping the implementation dead simple. It's a win.

Having said that, I have a pretty trivial LINQ-based tweak to Rufus' answer which addresses your other requirement (which is for the same object, for a given task, it should be executed in series). The solution works provided that the following assumptions hold:

The order in which the tasks are executed is not significant.
The work to be performed (all combinations of task x object) is known in advance and cannot change.
(Sorry for stating the obvious) The work which you want to parallelise can be parallelised - i.e. there are no shared resources / side-effects are completely isolated.

With those assumptions in mind, consider the following:

// Cartesian product of the two sets (*objects* and *tasks*).
var workItems = objects.SelectMany(
    o => tasks.Select(t => new { Object = o, Task = t })
);

// Group *work items* and materialise *work item groups*.
var workItemGroups = workItems
    .GroupBy(i => i, (key, items) => items.ToArray())
    .ToArray();

Parallel.ForEach(workItemGroups, workItemGroup =>
{
    // Execute non-unique *task* x *object*
    // combinations sequentially.
    foreach (var workItem in workItemGroup)
    {
        workItem.Task.Execute(workItem.Object);
    }
});

Note that I am not limiting the degree of parallelism in Parallel.ForEach. Since all work is CPU-bound, it will work out the best number of threads on its own.

Collectives™ on Stack Overflow

Hashing on multiple keys : for task execution In Multi threaded environment

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related