Convert a bytes array into JSON format

Question

I want to parse a bytes string in JSON format to convert it into python objects. This is the source I have:

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

And this is the desired outcome I want to have:

[{
"Date": "2016-05-21T21:35:40Z",
"CreationDate": "2012-05-05",
"LogoType": "png",
"Ref": 164611595,
"Classes": [
  "Email addresses",
  "Passwords"
],
"Link": "http://some_link.com"}]

First, I converted the bytes to string:

my_new_string_value = my_bytes_value.decode("utf-8")

but when I try to invoke loads to parse it as JSON:

my_json = json.loads(my_new_string_value)

I get this error:

json.decoder.JSONDecodeError: Expecting value: line 1 column 174 (char 173)

I've converted the bytes to string by using .decode("utf-8") but when I try to convert the string to JOSN i get this error json.decoder.JSONDecodeError: Expecting value:line 1 column 174 (char 173) — Merouane Benthameur
– Merouane Benthameur, Commented Oct 15, 2016 at 13:49
Can you update your question with the relevant code and print out the decoded string? — OneCricketeer
– OneCricketeer, Commented Oct 15, 2016 at 13:50
@MerouaneBenthameur The reason it fails is because the string you have is not JSON. The most obvious thing is that JSON uses ", not '. — freakish
– freakish, Commented Oct 15, 2016 at 13:58

PM 2Ring · Accepted Answer · 2020-06-22 16:50:57Z

216

Your bytes object is almost JSON, but it's using single quotes instead of double quotes, and it needs to be a string. So one way to fix it is to decode the bytes to str and replace the quotes. Another option is to use ast.literal_eval; see below for details. If you want to print the result or save it to a file as valid JSON you can load the JSON to a Python list and then dump it out. Eg,

import json

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

# Decode UTF-8 bytes to Unicode, and convert single quotes 
# to double quotes to make it valid JSON
my_json = my_bytes_value.decode('utf8').replace("'", '"')
print(my_json)
print('- ' * 20)

# Load the JSON to a Python list & dump it back out as formatted JSON
data = json.loads(my_json)
s = json.dumps(data, indent=4, sort_keys=True)
print(s)

output

[{"Date": "2016-05-21T21:35:40Z", "CreationDate": "2012-05-05", "LogoType": "png", "Ref": 164611595, "Classe": ["Email addresses", "Passwords"],"Link":"http://some_link.com"}]
- - - - - - - - - - - - - - - - - - - - 
[
    {
        "Classe": [
            "Email addresses",
            "Passwords"
        ],
        "CreationDate": "2012-05-05",
        "Date": "2016-05-21T21:35:40Z",
        "Link": "http://some_link.com",
        "LogoType": "png",
        "Ref": 164611595
    }
]

As Antti Haapala mentions in the comments, we can use ast.literal_eval to convert my_bytes_value to a Python list, once we've decoded it to a string.

from ast import literal_eval
import json

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

data = literal_eval(my_bytes_value.decode('utf8'))
print(data)
print('- ' * 20)

s = json.dumps(data, indent=4, sort_keys=True)
print(s)

Generally, this problem arises because someone has saved data by printing its Python repr instead of using the json module to create proper JSON data. If it's possible, it's better to fix that problem so that proper JSON data is created in the first place.

edited Jun 22, 2020 at 16:50

answered Oct 15, 2016 at 14:22

PM 2Ring

55.6k6 gold badges96 silver badges203 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Antti Haapala Over a year ago

I don't believe it is a JSON string, rather a Python repr, so use literal_eval instead

PM 2Ring Over a year ago

BTW, if you want to analyze or traverse a complicated JSON structure please see stackoverflow.com/a/52414034/4014959 & stackoverflow.com/a/41778581/4014959

Vercingatorix Over a year ago

Re: Generally this problem arises ... proper JSON data is created: Bytes data is apparently not JSON serializable: json.dumps(b'\0x41\x45') -> TypeError: Object of type bytes is not JSON serializable

PM 2Ring Over a year ago

@Vercingatorix JSON is for serializing data that's ultimately composed of strings, numbers, and booleans (or null), it's not designed to cope with arbitrary binary data. But it's easy enough to transform such data into a JSON-friendly form, eg you can create a simple hex string with docs.python.org/3/library/stdtypes.html#bytes.hex Or you could optimize that slightly with docs.python.org/3/library/base64.html

Chaithanya Krishna · Accepted Answer · 2023-05-19 06:22:04Z

119

You can simply use,

import json

my_bytes_value = my_bytes_value.decode().replace("'", '"')
json.loads(my_bytes_value)

edited May 19, 2023 at 6:22

answered Oct 16, 2020 at 8:11

Chaithanya Krishna

1,4941 gold badge9 silver badges13 bronze badges

4 Comments

Hari Kishore Over a year ago

this should have green tick.

Pierre Monico Over a year ago

Not really. This answers the question in the title but omits that the example provided by OP has single quotes. json.loads(my_bytes_value) would throw a json.decoder.JSONDecodeError in this case.

Chaithanya Krishna Over a year ago

Yes, you are right @pierre-monico. I have updated the answer accordingly. Thanks for flagging.

Zhanwen Chen Over a year ago

For me, I had to remove .replace("'", '"')

Novikov · Accepted Answer · 2020-12-16 19:45:25Z

10

Python 3.5 + Use io module

import json
import io

my_bytes_value = b'[{\'Date\': \'2016-05-21T21:35:40Z\', \'CreationDate\': \'2012-05-05\', \'LogoType\': \'png\', \'Ref\': 164611595, \'Classe\': [\'Email addresses\', \'Passwords\'],\'Link\':\'http://some_link.com\'}]'

fix_bytes_value = my_bytes_value.replace(b"'", b'"')

my_json = json.load(io.BytesIO(fix_bytes_value))

answered Dec 16, 2020 at 19:45

Novikov

7607 silver badges18 bronze badges

1 Comment

Jason Rahm Over a year ago

thank you! this is the the least icky and most direct path, particularly if the json within the bytesarray is already properly formatted.

Suraj Rao · Accepted Answer · 2022-04-08 11:46:21Z

3

d = json.dumps(byte_str.decode('utf-8'))

edited Apr 8, 2022 at 11:46

Suraj Rao

29.7k11 gold badges96 silver badges104 bronze badges

answered Apr 8, 2022 at 11:43

Kwame

1,0973 gold badges21 silver badges38 bronze badges

Comments

Simon · Accepted Answer · 2016-10-15 13:55:48Z

2

To convert this bytesarray directly to json, you could first convert the bytesarray to a string with decode(), utf-8 is standard. Change the quotation markers.. The last step is to remove the " from the dumped string, to change the json object from string to list.

dumps(s.decode()).replace("'", '"')[1:-1]

answered Oct 15, 2016 at 13:55

Simon

4244 silver badges12 bronze badges

Comments

EMAI · Accepted Answer · 2022-02-28 15:55:49Z

2

Better solution is:

import json
byte_array_example = b'{"text": "\u0627\u06CC\u0646 \u06CC\u06A9 \u0645\u062A\u0646 \u062A\u0633\u062A\u06CC \u0641\u0627\u0631\u0633\u06CC \u0627\u0633\u062A."}'    
res = json.loads(byte_array_example.decode('unicode_escape'))
print(res)

result:

{'text': 'این یک متن تستی فارسی است.'}

decode by utf-8 cannot decode unicode characters. The right solution is uicode_escape

It is OK

answered Feb 28, 2022 at 15:55

EMAI

7428 silver badges15 bronze badges

Comments

Darkstar Dream · Accepted Answer · 2022-08-04 06:59:06Z

1

if you have a bytes object and want to store it in a JSON file, then you should first decode the byte object because JSON only has a few data types and raw byte data isn't one of them. It has arrays, decimal numbers, strings, and objects.

To decode a byte object you first have to know its encoding. For this, you can use

import chardet
encoding = chardet.detect(your_byte_object)['encoding']

then you can save this object to your json file like this

data = {"data": your_byte_object.decode(encoding)}
with open('request.txt', 'w') as file:
    json.dump(data, file)

answered Aug 4, 2022 at 6:59

Darkstar Dream

1,8992 gold badges16 silver badges33 bronze badges

Comments

Maso Mahboob · Accepted Answer · 2022-10-10 05:15:57Z

1

The most simple solution is to use the json function that comes with http request.

For example:

answered Oct 10, 2022 at 5:15

Maso Mahboob

1171 silver badge3 bronze badges

1 Comment

Ruben Driezen Over a year ago

I know this answer is 2 years old, but why would you ever share your code as an image instead of actual text? This is isn't copyable, takes longer to load, and ignores the styling preferences of everyone who reads this.

randomDev · Accepted Answer · 2024-01-12 11:20:09Z

1

json.loads(body.decode('utf-8'))

This will decode and convert it to a Python dictionary.

The body in the above code is from a fastapi libraries Request object. This object's body attribute value will be in bytes.

edited Jan 12, 2024 at 11:20

answered Jan 12, 2024 at 11:08

randomDev

3673 silver badges16 bronze badges

Collectives™ on Stack Overflow

Convert a bytes array into JSON format

9 Answers 9

4 Comments

4 Comments

1 Comment

Comments

Comments

Comments

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

4 Comments

4 Comments

1 Comment

Comments

Comments

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related