Execute vectorized function on multiple cores

Question

I have some function f(x,y) which I have vectorized with numpy.vectorize command. I have made some grid values of x and y for which I want the function to be evaluated. My program looks like this:

from numpy import vectorize,meshgrid, linspace

@vectorize
def f(x,y):
    pass

x = linspace(0,10)
y = linspace(0,10)
X, Y = meshgrid(x, y)
Z = f(X,Y)

When I look at the system monitor on evaluation time (for example with htop on Ubuntu) I see that only one core is being used. What are the options to get most juice of system on such computation?

what does your f(x,y) do? instead of using @vectorize, it is better to implement your function so that it can handle ndarray arguments, since numpy.vectorize is bad — usethedeathstar
– usethedeathstar, Commented Apr 29, 2014 at 13:23
The python interpreter runs on only one core, though numpy can use a multi-core setup (see e.g. stackoverflow.com/questions/5991014/numpy-on-multicore-hardware). A simple way to make use of multiple cores is to use something like the multiprocessing module. — jmetz
– jmetz, Commented Apr 29, 2014 at 13:23
Generally the f(x,y) integrates some other function g(T,x,y) along T with scipy.integrate.quad. Why numpy.vectorize is bad? It makes coding much easier! — Jānis Erdmanis
– Jānis Erdmanis, Commented Apr 29, 2014 at 15:45
Okay how to implement vectorize function in order to use multiple cores? — Jānis Erdmanis
– Jānis Erdmanis, Commented Apr 29, 2014 at 15:52

Adobe · Accepted Answer · 2015-09-15 14:07:42Z

3

Using numpy.vectorize is not usually a good idea. It doesn't parallelize anything, and in fact the performance is generally not even very good because of the way it works (as explained in its documentation). So instead, you need to work on making your f(x,y) function vectorized internally, or implement it as a module in C or C++ or Fortran etc.

If you want to truly use multiple cores in Python, your best bet is either to use the threading module if your function is implemented in C, or multiprocessing if you're trying to run native Python (including NumPy).

edited Sep 15, 2015 at 14:07

Adobe

13.5k12 gold badges94 silver badges132 bronze badges

answered Apr 29, 2014 at 13:28

John Zwinck

252k44 gold badges346 silver badges459 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ev-br Over a year ago

(+1 for what's in the answer.) If numexpr can handle the function f(x, y), that can be the easiest way of using several cores.

Collectives™ on Stack Overflow

Execute vectorized function on multiple cores

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related