7

BACKGROUND

The issue I'm working with is as follows:

  • Within the context of an experiment I am designing for my research, I produce a large number of large (length 4M) arrays which are somewhat sparse, and thereby could be stored as scipy.sparse.lil_matrix instances, or simply as scipy.array instances (the space gain/loss isn't the issue here).

  • Each of these arrays must be paired with a string (namely a word) for the data to make sense, as they are semantic vectors representing the meaning of that string. I need to preserve this pairing.

  • The vectors for each word in a list are built one-by-one, and stored to disk before moving on to the next word.

  • They must be stored to disk in a manner which could be then retrieved with dictionary-like syntax. For example if all the words are stored in a DB-like file, I need to be able to open this file and do things like vector = wordDB[word].

CURRENT APPROACH

What I'm currently doing:

  • Using shelve to open a shelf named wordDB

  • Each time the vector (currently using lil_matrix from scipy.sparse) for a word is built, storing the vector in the shelf: wordDB[word] = vector

  • When I need to use the vectors during the evaluation, I'll do the reverse: open the shelf, and then recall vectors by doing vector = wordDB[word] for each word, as they are needed, so that not all the vectors need be held in RAM (which would be impossible).

The above 'solution' fits my needs in terms of solving the problem as specified. The issue is simply that when I wish to use this method to build and store vectors for a large amount of words, I simply run out of disk space.

This is, as far as I can tell, because shelve pickles the data being stored, which is not an efficient way of storing large arrays, thus rendering this storage problem intractable with shelve for the number of words I need to deal with.

PROBLEM

The question is thus: is there a way of serializing my set of arrays which will:

  1. Save the arrays themselves in compressed binary format akin to the .npy files generated by scipy.save?

  2. Meet my requirement that the data be readable from disk as a dictionary, maintaining the association between words and arrays?

Improve this question
5
  • Just to make sure, you are using protocol 2 for pickling (in your shelf), correct? docs.python.org/library/pickle.html#data-stream-format If you do not specify a protocol, it defaults to 0, which is the least efficient. Commented Mar 16, 2011 at 19:36
  • As a matter of fact... I didn't know that. I sort of assumed it would default to the most practical. I'll just test whether or not there's a significan space gain on my data with protocol 2. Thanks for pointing this out! Commented Mar 17, 2011 at 2:50
  • I also assumed it used the most favorable protocol by default and got burned before, which is why I thought I'd ask. I suppose the default is protocol 0, instead, for legacy reasons. Commented Mar 17, 2011 at 14:38
  • As shown in my answer, you can use -1 or cPickle.HIGHEST_PROTOCOL to always use the most newest (and I assume going forward, most efficient) pickling protocol. Also, you'll get better performance out of cPickle vs Pickle. Commented Mar 17, 2011 at 15:42
  • Okay, using protocol two halves the storage space needed. That's sadly not enough, but it's good progress. I'll have to look at this hdf5 stuff... Commented Mar 18, 2011 at 1:16

4 Answers 4

Reset to default
5

as JoshAdel already suggested, I would go for HDF5, the simplest way is to use h5py:

http://h5py.alfven.org/

you can attach several attributes to an array with a dictionary like sintax:

dset.attrs["Name"] = "My Dataset"

where dset is your dataset which can be sliced exactly as a numpy array, but in the background it does not load all the array into memory.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

This is the solution I went for, except using pytables. Thanks!
2

I would suggest to use scipy.save and have an dictionnary between the word and the name of the files.

Improve this answer

1 Comment

This might ultimately be the solution I end up doing but it feels somewhat inelegant, and I'm keen to see if there are existing solutions before reinventing the wheel. Good suggestion, though. Thanks.
2

Have you tried just using cPickle to pickle the dictionary directly using:

import cPickle
DD = dict()
f = open('testfile.pkl','wb')
cPickle.dump(DD,f,-1)
f.close()

Alternatively, I would just save the vectors in a large multidimensional array using hdf5 or netcdf if necessary since this allows you to open a large array without bringing it all into memory at once and then get slices as needed. You can then associate the words as an additional group in the netcdf4/hdf5 file and use the common indices to quickly associate the appropriate slice from each group, or just name the group the word and then have the data be the vector. You'd have to play around with which is more efficient.

http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4-module.html

Pytables also might be a useful storage layer on top of HDF5:

http://www.pytables.org

Improve this answer

Comments

2

Avoid using shelve, it's bug ridden and has cross-platform issues.

The memory issue, however, has nothing to do with shelve. Numpy arrays provide efficient implementation of the pickle protocol and there is little memory overhead to cPickle.dumps(protocol=-1), compared to binary .npy (only the extra headers in pickle, basically).

So if binary/pickle isn't enough, you'll have to go for compression. Have a look at pytables or h5py (difference between the two).

If specifying the binary protocol in pickle is enough, you can consider something more lightweight than hdf5: check out sqlitedict for a replacement of shelve. It has no additional dependencies.

Improve this answer

2 Comments

shelve is bug-ridden --- can you explain this a bit more?
sqlitedict seems superior with similar dictionary semantics. curious about the status of shelve bugs as well, as I've had issues with it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.