Parallelizing multiplication of vectors-like computation in python

Question

I have got a chunk of code like

for i in range(0, len(a))
    b[i] = func(a[i])

where a and b are arrays of the same length, a is given (and big), func is some function that has a lot of local variables but does not use any global variables.

I would like to distribute computations of func across several CPUs. Presumably I need to use multiprocessing module, but I have not found any relevant examples. Could you help? Thanks.

Which Python interpreter you are using? Note that if using CPython, you need to be aware of GIL: wiki.python.org/moin/GlobalInterpreterLock — jsalonen
– jsalonen, Commented Jun 20, 2012 at 11:37
If you are using linux, you can use fork. ikharn.blogspot.in/2012/04/multiprocessing-in-python.html — Netro
– Netro, Commented Jun 20, 2012 at 11:48
@Nikhil: Using fork is problematic because it doesn't have an easy way of getting results back to the parent process. — Gabe
– Gabe, Commented Jun 21, 2012 at 2:33

jfs · Accepted Answer · 2012-06-20 11:58:42Z

3

See the very first code example in the multiprocessing docs:

from multiprocessing import Pool

# you could define `func`, `a` here

if __name__=="__main__":
    p = Pool() # use all available CPU cores
    b = p.map(func, a)

edited Jun 20, 2012 at 11:58

answered Jun 20, 2012 at 11:53

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Maria Zverina · Accepted Answer · 2012-06-20 13:17:27Z

1

Use process pool. You can see a full sample in my github: https://github.com/mariazverina/codejam/blob/master/src/main.py

from multiprocessing import Pool

p = Pool(4)  # set to number of cores
b = p.map(func, a)

edited Jun 20, 2012 at 13:17

answered Jun 20, 2012 at 11:55

Maria Zverina

11.2k3 gold badges47 silver badges62 bronze badges

2 Comments

Maria Zverina Over a year ago

good point - brain -> fingers malfunction - updated explanation above

jfs Over a year ago

on Windows if __name__=="__main__" is mandatory when using multiprocessing.

Collectives™ on Stack Overflow

Parallelizing multiplication of vectors-like computation in python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related