Understanding dask array chunks

Question

I am trying to learn how array chunking works by reading the docs here.

Below is an output from a Python session where I try to reproduce the examples.

In [1]: import numpy as np

In [2]: npa = np.array([
   ...: [1, 2, 3, 4, 5, 6],
   ...: [7, 8, 9, 0, 1, 2],
   ...: [3, 4, 5, 6, 7, 8],
   ...: [9, 0, 1, 2, 3, 4],
   ...: [5, 6, 7, 8, 9, 0],
   ...: [1, 2, 3, 4, 5, 6]
   ...: ])

In [3]: import dask.array as da

In [4]: a = da.from_array(npa, chunks=3)

In [5]: a
Out[5]: dask.array<array, shape=(6, 6), dtype=int64, chunksize=(3, 3), chunktype=numpy.ndarray>

I would expect each chunk(/block?) of a to have shape (3, 3) since that is what I specified in the chunks parameter and what the example in the docs seems to suggest.

However, when I read out the first block, it has a shape of (3, 6).

In [6]: a.blocks[0]
Out[6]: dask.array<blocks, shape=(3, 6), dtype=int64, chunksize=(3, 3), chunktype=numpy.ndarray>

And, as expected given the shape, I can only read out two blocks. When I read out the third block an IndexError is raised.

In [7]: a.blocks[2]
...
IndexError: Index is not smaller than dimension 2 >= 2

I would expect there to be four 3x3 blocks, not two 3x6 blocks.

What am I not understanding about how array chunking works in dask?

Agreed, this does seem odd. Maybe something has changed in github.com/dask/dask/blob/master/dask/array/core.py#L2357 ? I'd recommend filing an issue. In the case you've outlined above dask builds two arrays of (3,6) which is strange given the docs — quasiben
– quasiben, Commented Jul 28, 2020 at 17:32

Josh Reback · Accepted Answer · 2020-08-04 12:14:12Z

1

Answered here.

Your chunks/blocks are in two dimensions, so you can index them in two dimensions

answered Aug 4, 2020 at 12:14

Josh Reback

5996 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Understanding dask array chunks

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related