Parallelize loop over numpy rows

Question

I need to apply the same function onto every row in a numpy array and store the result again in a numpy array.

# states will contain results of function applied to a row in array
states = np.empty_like(array)

for i, ar in enumerate(array):
    states[i] = function(ar, *args)

# do some other stuff on states

function does some non trivial filtering of my data and returns an array when the conditions are True and when they are False. function can either be pure python or cython compiled. The filtering operations on the rows are complicated and can depend on previous values in the row, this means I can't operate on the whole array in an element-by-element fashion

Is there a way to do something like this in dask for example?

It still doesn't make sense. Where is i coming from? Are you trying to call enumerate? — Neil G
– Neil G, Commented Sep 28, 2015 at 8:29
Your function takes only current row or it can take any other row also? — hellpanderr
– hellpanderr, Commented Sep 28, 2015 at 8:54
The function accepts any 1D numpy array. It's doesn't care where that array came from. — Max Linke
– Max Linke, Commented Sep 28, 2015 at 13:46

MRocklin · Accepted Answer · 2015-09-28 15:56:18Z

7

Dask solution

You could do with with dask.array by chunking the array by row, calling map_blocks, then computing the result

ar = ...
x = da.from_array(ar, chunks=(1, arr.shape[1]))
x.map_blocks(function, *args)
states = x.compute()

By default this will use threads, you can use processes in the following way

from dask.multiprocessing import get
states = x.compute(get=get)

Pool solution

However dask is probably overkill for embarrassingly parallel computations like this, you could get by with a threadpool

from multiprocessing.pool import ThreadPool
pool = ThreadPool()

ar = ...
states = np.empty_like(array)

def f(i):
    states[i] = function(ar[i], *args)

pool.map(f, range(len(ar)))

And you could switch to processes with the following change

from multiprocessing import Pool
pool = Pool()

answered Sep 28, 2015 at 15:56

MRocklin

57.5k29 gold badges176 silver badges245 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Neil G · Accepted Answer · 2015-09-28 06:31:19Z

0

Turn your function into a universal function: http://docs.scipy.org/doc/numpy/reference/ufuncs.html.

Then: states = function(array, *args).

answered Sep 28, 2015 at 6:31

Neil G

33.6k44 gold badges162 silver badges267 bronze badges

2 Comments

Max Linke Over a year ago

I can't operate on the array in an element-by-element fashion. The filtering of a single row depends on previous values.

Neil G Over a year ago

@kain88: You don't use i properly in your for loop and you don't seem to pass it to the function. It doesn't make any sense.

Collectives™ on Stack Overflow

Parallelize loop over numpy rows

2 Answers 2

Dask solution

Pool solution

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Dask solution

Pool solution

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related