5

Is there a good way to load a bytes object that is represented as a string, so it can be unpickled?

Basic Example

Here is a dumb example:

import pickle

mydict = { 'a': 1111, 'b': 2222 }
string_of_bytes_obj = str(pickle.dumps(mydict)) # Deliberate string representation for this quick example.

unpickled_dict = pickle.loads(string_of_bytes_obj) # ERROR!  Loads takes bytes-like object and not string.

Attempt at a Solution

One solution is of course to eval the string:

unpickled_dict = pickle.loads(eval(string_of_bytes_obj))

But, seems wrong to eval, especially when the strings might be coming over a network or from a file.

...

Any suggestions for a better solution?

Thanks!

Improve this question
6
  • I think it is much better just not to convert bytes to str. Is there any way to avoid that conversion? Commented Oct 19, 2018 at 11:02
  • 1
    @soon I'm just processing a file from someone else... would definitely be better to change their file to use binary data... but right now, I can't avoid the string conversion. Commented Oct 19, 2018 at 11:09
  • 1
    It's not a good idea to use pickle with network data btw: zopatista.com/plone/2007/11/09/one-cookie-please Commented Oct 19, 2018 at 11:12
  • Do you know what the encoding of the file is? If you do then you just have to str.encode the string to unpickle. If you don't you need to guess the encoding first. Commented Oct 19, 2018 at 11:31
  • @JacquesGaudin not sure I understand. What would be the encoding of the string? I know the file is utf-8, but I think all that gives me is a way to get the string representation in a known encoding, and I would still need to eval... or do you mean something like the answer provided by @Farhan.K? Commented Oct 19, 2018 at 12:40

4 Answers 4

Reset to default
7

For a safety concern you can use ast.literal_eval instead of eval:

>>> import ast
>>> pickle.loads(ast.literal_eval(string_of_bytes_obj))
{'b': 2222, 'a': 1111}
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks. Great answer. Safer evaluation, just needs some extra error handling.
4

You can use encoding="latin1" as an argument to str and then use bytes to convert back:

import pickle

mydict = { 'a': 1111, 'b': 2222 }
string_of_bytes_obj = str(pickle.dumps(mydict), encoding="latin1")

unpickled_dict = pickle.loads(bytes(string_of_bytes_obj, "latin1"))

Output:

>>> print(unpickled_dict)
{'a': 1111, 'b': 2222}
Improve this answer

1 Comment

Why does this work?! I just spent 2 hours trying to figure this out. No answers anywhere else!
0

Is there a reason you need to have it as a str? If you're just writing it to file, you can 'wb' instead of 'w'. (https://pythontips.com/2013/08/02/what-is-pickle-in-python/)

import pickle

mydict = { 'a': 1111, 'b': 2222 }
dumped = pickle.dumps(mydict)
string_of_bytes_obj = str(dumped) # Deliberate string representation for this quick example. 

unpickled_dict = pickle.loads(dumped)
Improve this answer

1 Comment

Unfortunately, not my file, I'm just processing what I'm given!
-1

First of all i wouldn't use pickles to serialize data. instead use Json.

my solution with pickles 

import pickle

mydict = { 'a': 1111, 'b': 2222 }
string_of_bytes_obj = pickle.dumps(mydict) # Deliberate string representation for this quick example.
print(string_of_bytes_obj)
unpickled_dict = pickle.loads(string_of_bytes_obj)
print(unpickled_dict)

BUT with json 

import json

mydict = { 'a': 1111, 'b': 2222 }
string_of_bytes_obj = json.dumps(mydict) 
print(string_of_bytes_obj)
unpickled_dict = json.loads(string_of_bytes_obj)
print(unpickled_dict)

I highly recommend you to use json to serialize your data

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.