Running Multiple Process At The Same Time And Returning the Results

Question

I want to execute a function at the same time with different arguments and also have to get the return values back in my main. A pseudocode for my problem is

def Function(data):
      print("Function Running With Values" , data[0],"and",data[1],"At timestamp : " ,time.time())
      return data[0]+data[1]


Results= []
Values= [ [1,2],[2,3],[4,5],[1,4]]
#calling function for all Values and return the result in results variable in sequence.

Timus · Accepted Answer · 2020-11-13 14:47:44Z

1

I'm not sure what you mean with "at the same time":

If a sequential processing is fine this

Results = list(map(Function, Values))
print(Results)

or more pythonic a list comprehension

Results = [Function(value) for value in Values]
print(Results)

gives you the following output

Function Running With Values 1 and 2 At timestamp :  1605276375.4642859
Function Running With Values 2 and 3 At timestamp :  1605276375.4645345
Function Running With Values 4 and 5 At timestamp :  1605276375.4647174
Function Running With Values 1 and 4 At timestamp :  1605276375.4648669
[3, 5, 9, 5]

If you actually want multiprocessing then this

import multiprocessing as mp

with mp.Pool() as p:
    Results = list(p.map(Function, Values))

print(Results)

or this

from concurrent.futures import ProcessPoolExecutor

with ProcessPoolExecutor() as p:
    Results = list(p.map(Function, Values))

print(Results)

gives you output like

Function Running With Values 1 and 2 At timestamp :  1605276375.4532914
Function Running With Values 4 and 5 At timestamp :  1605276375.4547572
Function Running With Values 2 and 3 At timestamp :  1605276375.4549458
Function Running With Values 1 and 4 At timestamp :  1605276375.456188
[3, 5, 9, 5]

If you want multiprocessing then you should look a bit deeper into it to make sure nothing goes wrong and the processing is indeed faster. But your example is a classic MapReduce scenario that should work fine.

Is that what you were looking for?

edited Nov 13, 2020 at 14:47

answered Nov 13, 2020 at 14:28

Timus

11.4k5 gold badges20 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Tiger Strom Over a year ago

Thank You.! Solved my problem perfectly..! by 'at the same time' I mean't multiprocessing only :)

Timus Over a year ago

@TigerStrom Then sorry for the other stuff! Some additional points: You need to play around a bit to achieve the optimal speed. Both, mp.Pool and ProcessPoolExecutor, allow for specifiying the max. number of process (workers, ...) that should run concurrently. The standard is the number of cpus (cores) your machine has, but that isn't always the best choice. .map has an optional argument chunksize. I found that setting it to something like length of the argument list divided by the specified number of processes often the best choice. ...

Collectives™ on Stack Overflow

Running Multiple Process At The Same Time And Returning the Results

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related