Saving List of Numpy 2D arrays using numpy.save (the arrays together are jagged)

Question

I have a large image dataset. When I use the images, I have several components--a mirrored image, a regular image, an eigenvector matrix and an eigenvalue vector.

I would like to store it like:

training_sunsets_data = [cropped_training_sunsets,
                         mirrored_training_sunsets,
                         rgb_cov_eigvec_training_sunsets,
                         rgb_cov_eigval_training_sunsets]

np.save('training_sunsets_data',training_sunsets_data)

And as I was writing this I was testing it (because I was sure it would fail), and the strangest thing happened when I did this: it worked.

Further, when I loaded it back up into the code, it was type ndarray, but it is a jagged array.

How is this possible if numpy does not allow jagged multidimensional arrays? Did I just find a backdoor way to create a jagged array in numpy?

Can you post the jagged ndarray you got from loading the file? I am curious to see what it looks like. — gdlmx
– gdlmx, Commented Apr 22, 2016 at 23:37
Look at np.savez. That saves each array by name in a file, and collects them in a zip archive. np.load handles that kind of archive. — hpaulj
– hpaulj, Commented Apr 22, 2016 at 23:49
There's a very good answer here: stackoverflow.com/questions/48603119/… — mLstudent33
– mLstudent33, Commented Sep 4, 2019 at 4:42

gdlmx · Accepted Answer · 2016-04-22 23:49:21Z

13

After testing on my machine:

  import numpy as np
  np.save('testnp.npy', [[2,3,4],[1,2]])
  np.load('testnp.npy')
  #   array([[2, 3, 4], [1, 2]], dtype=object)

As shown in the example code, the loaded object is of type ndarray, but its data type is object. That means, np.save store an array of python objects, which can be anything. According to the documentation, it seems to use python pickle to pack those objects.

So you didn't find a backdoor, it behaves just as expected.

edited Apr 22, 2016 at 23:49

answered Apr 22, 2016 at 23:46

gdlmx

6,8691 gold badge25 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mdurant Over a year ago

...and you could as well use pickle to store sets/sequences of arrays, if you wish.

Dan Erez · Accepted Answer · 2019-05-28 14:04:33Z

2

np.savez() would work in your situation. save each as a variable.

answered May 28, 2019 at 14:04

Dan Erez

1,54418 silver badges16 bronze badges

Comments

kpie · Accepted Answer · 2016-04-22 23:33:35Z

0

So to see what you are getting at lets runs some code.

>>> a =[np.array([[1,2,3],[4,5,6]]),np.array([[1,2],[3,4]])]
>>> type(a)
<type 'list'>
>>> np.array(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: could not broadcast input array from shape (2,3) into shape (2)

We see here that we are perfectly able to make a list of np.arrays of different dimensions. We cannot however cast that list into a np.array.

I suspect based on your syntax that you are saving a list, and loading a list maintaining the type np.array for each element in the list.

answered Apr 22, 2016 at 23:33

kpie

11.2k7 gold badges33 silver badges53 bronze badges

Collectives™ on Stack Overflow

Saving List of Numpy 2D arrays using numpy.save (the arrays together are jagged)

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related