How dask array chunking works for three or more dimension?

Question

Consider the following code snippet where I am trying to create a dask array with asymmetric but repeated blocks of size (3, 4, 5):

import numpy as np
import dask.array as da
a = np.random.randint(0, 9, (3, 12, 10))
d = da.from_array(a, chunks=(3, 4, 5))

The above snippet does not throw any error / warning. But When I try to do the following:

r = d.map_blocks(np.sum)
out = r.compute()

It throws the error below:

python3.7/site-packages/dask/array/core.py in <listcomp>(.0)
   4099 
   4100     while isinstance(arrays, (list, tuple)):
-> 4101         result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
   4102         arrays = arrays[0]
   4103         dim += 1

IndexError: tuple index out of range

What am I doing wrong?

Ke Zhang · Accepted Answer · 2020-02-15 19:37:11Z

2

If you check map_block docs here. Under chunks parameter description, it said:

"If not provided the resulting array is assumed to have the same block structure as the first input array."

So your sum function should be something like this:

def compute_block_sum(block):
    return np.array([[np.sum(block)]])[:,None]

Then you do

d.map_blocks(compute_block_sum).compute()

edited Feb 15, 2020 at 19:37

answered Feb 15, 2020 at 19:27

Ke Zhang

9981 gold badge11 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How dask array chunking works for three or more dimension?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related