Unpickling from converted string in python/numpy

Question

I have a ton of numpy ndarrays that are stored picked to strings. That may have been a poor design choice but it's what I did, and now the picked strings seem to have been converted or something along the way, when I try to unpickle I notice they are of type str and I get the following error:

TypeError: 'str' does not support the buffer interface

when I invoke

numpy.loads(bin_str)

Where bin_str is the thing I'm trying to unpickle. If I print out bin_strit looks like

b'\x80\x02cnumpy.core.multiarray\n_reconstruct\nq\x00cnumpy\nndarray\nq\x01K\x00\x85q\x02c_codecs\nencode\nq\x03X\x01\x00\x00\ ...

continuing for some time, so the info seems to be there, I'm just not quite sure how to convert it into whatever string format numpy/pickle need. On a whim I tried

numpy.loads( bytearray(bin_str, encoding='utf-8') )

and

numpy.loads( bin_str.encode() )

which both throw an error _pickle.UnpicklingError: unpickling stack underflow. Any ideas?

PS: I'm on python 3.3.2 and numpy 1.7.1

Edit

I discovered that if I do the following:

open('temp.txt', 'wb').write(...)
return numpy.load( 'temp.txt' )

I get back my array, and ... denotes copying and pasting the output of print(bin_str) from another window. I've tried writing bin_str to a file directly to unpickle but that doesn't work, it complains that TypeError: 'str' does not support the buffer interface. A few sane ways of converting bin_str to something that can be written directly to a binary file result in pickle errors when trying to read it back.

Edit 2 So I guess what's happened is that my binary pickle string ended up encoded inside of a normal string, something like:

"b'pickle'"

which is unfortunate and I haven't figured out how to deal with that, except this ridiculous and convoluted way to get it back:

open('temp.py', 'w').write('foo = ' + bin_str)
from temp import foo
numpy.loads( foo )

This seems like a very shameful solution to the problem, so please give me a better one!

Blckknght · Accepted Answer · 2013-10-28 22:33:05Z

2

It sounds like your saved strings are the reprs of the original bytes instances returned by your pickling code. That's a bit unfortunate, but not too bad. repr is intended to return a "machine friendly" representation of an object, and it can often be reversed by using eval:

import numpy as np
import pickle

# this part has already happened
orig_obj = np.array([1,2,3])
orig_pickle = pickle.dumps(orig_obj)
saved_str = repr(orig_pickle)     # this was a mistake, but it's already done

# this is what you need to do to get something equivalent to orig_obj back
reconstructed_pickle = eval(saved_str)
reconstructed_obj = pickle.loads(reconstructed_pickle)

# test
if np.all(reconstructed_obj == orig_obj):
    print("It worked!")

Obligatory note that using eval can be dangerous: Be aware that eval can run any Python code it wants, so don't call it with untrusted data. However, pickle data has the same risks (a malicious Pickle string can run arbitrary code upon unpickling), so you're not losing much safety in this situation. I'm guessing that you trust your data in this case anyway.

answered Oct 28, 2013 at 22:33

Blckknght

106k11 gold badges135 silver badges188 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alex Z Over a year ago

Thanks! This is working perfectly. I had no idea eval/exec were in the language, heh.

Collectives™ on Stack Overflow

Unpickling from converted string in python/numpy

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related