Numpy - Stacked memory view of two 1D arrays

Question

I know that I can do the following:

import numpy as np
c = np.random.randn(20, 2)
a = c[:, 0]
b = c[:, 1]

Here, a and b are pointers to c's first and second column respectively. Modifying a or b will change c (same reciprocally).

However, what I want to achieve is exactly the opposite. I want to create a 2D memory view where each column (or row) will point to a memory of a different 1D array. Assume that I already have two 1D arrays, is it possible to create a 2D view to these arrays where each row/column points to each of them?

I can create c from a and b in the following way:

c = np.c_[a, b]

However, this copies a's and b memory onto c. Can I just somehow create c as 'view' of [a b], where, by modifying an element of c this reflects in the respective a or b 1D array?

hpaulj · Accepted Answer · 2015-02-14 18:24:04Z

I don't think it is possible.

In your first example, the values of the a and b views are interwoven, as can be seen from this variation:

In [51]: c=np.arange(10).reshape(5,2)
In [52]: a, b = c[:,0], c[:,1]
In [53]: a
Out[53]: array([0, 2, 4, 6, 8])
In [54]: c.flatten()
Out[54]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

The data buffer for c and a start at the same memory point; b starts at 4 bytes into that buffer.

In [55]: c.__array_interface__
Out[55]: 
{'strides': None,
 'data': (172552624, False),...}

In [56]: a.__array_interface__
Out[56]: 
{'strides': (8,),
 'data': (172552624, False),...}

In [57]: b.__array_interface__
Out[57]: 
{'strides': (8,),
 'data': (172552628, False),...}

Even if the a,b split were by rows, b would start just further along in the same shared data buffer.

From the .flags we see that c is C-contiguous, b is not. But b values are accessed with constant strides in that shared data buffer.

When a and b are created separately, their data buffers are entirely separate. The numpy striding mechanism cannot step back and forth between these two data buffers. A 2d composite of a and b has to work with its own data buffer.

I can imagine writing a class that ends up looking like what you want. The indexing_tricks file that defines np.c_ might give you ideas (e.g. a class with a custom __getitem__ method). But it wouldn't have the speed advantages of a regular 2d array. And it might be hard to implement all of the ndarray functionality.

Jaime · Accepted Answer · 2015-02-14 19:13:18Z

3

While @hpaulj's answer is the correct one, for your particular case, and more as an exercise in understanding numpy memory layout than as anything with practical applications, here's how you can get a view of two 1-D arrays as columns of a common array:

>>> from numpy.lib.stride_tricks import as_strided
>>> a = np.arange(10)
>>> b = np.arange(20, 30)
>>> col_stride = (b.__array_interface__['data'][0] -
                  a.__array_interface__['data'][0])
>>> c = as_strided(a, shape=(10, 2), strides=(a.strides[0], col_stride))
>>> c
array([[ 0, 20],
       [ 1, 21],
       [ 2, 22],
       [ 3, 23],
       [ 4, 24],
       [ 5, 25],
       [ 6, 26],
       [ 7, 27],
       [ 8, 28],
       [ 9, 29]])
>>> c[4, 1] = 0
>>> c[6, 0] = 0
>>> a
array([0, 1, 2, 3, 4, 5, 0, 7, 8, 9])
>>> b
array([20, 21, 22, 23,  0, 25, 26, 27, 28, 29])

There are many things that can go wrong here, mainly that the array b has not had its reference count increased, so if you delete it its memory will be released, but the view will still be accessing it. It can also not be extended to more than two 1-D arrays, and requires that both 1-D arrays have the same stride.

Of course, just because you can do it doesn't mean you should do it! And you should definitely not do this.

answered Feb 14, 2015 at 19:13

Jaime

67.7k19 gold badges128 silver badges164 bronze badges

1 Comment

Imanol Luengo Over a year ago

Thank you for the example. I understand this is not ideal, but I guess it could work in a 'controlled' environment (e.g. class) when you can assure b's existence. Anyway, thank you both for the explanations.

Collectives™ on Stack Overflow

Numpy - Stacked memory view of two 1D arrays

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related