How to convert raw javascript object to a dictionary?

Question

When screen-scraping some website, I extract data from <script> tags.
The data I get is not in standard JSON format. I cannot use json.loads().

# from
js_obj = '{x:1, y:2, z:3}'

# to
py_obj = {'x':1, 'y':2, 'z':3}

Currently, I use regex to transform the raw data to JSON format.
But I feel pretty bad when I encounter complicated data structure.

Do you have some better solutions?

@HuuNguyen I want to parse Plain old javascript data structure to python object. — kev
– kev, Commented Jun 4, 2014 at 1:32
Oh I didn't see that js_obj didn't have quotes around the keys. How complicated would your data structures get? It's hard to suggest anything without knowing the cases you're trying to solve for. — huu
– huu, Commented Jun 4, 2014 at 1:34
there are similar questions on SO already: stackoverflow.com/a/10057449/384442 none of them is offering any ready to use solution — RomanI
– RomanI, Commented Jun 4, 2014 at 1:49

Benjamin Loison · Accepted Answer · 2023-08-21 23:18:03Z

62

`demjson.decode()`

import demjson

# from
js_obj = '{x:1, y:2, z:3}'

# to
py_obj = demjson.decode(js_obj)

`chompjs.parse_js_object()`

import chompjs

# from
js_obj = '{x:1, y:2, z:3}'

# to
py_obj = chompjs.parse_js_object(js_obj)

`jsonnet.evaluate_snippet()`

import json, _jsonnet

# from
js_obj = '{x:1, y:2, z:3}'

# to
py_obj = json.loads(_jsonnet.evaluate_snippet('snippet', js_obj))

`ast.literal_eval()`

import ast

# from
js_obj = "{'x':1, 'y':2, 'z':3}"

# to
py_obj = ast.literal_eval(js_obj)

edited Aug 21, 2023 at 23:18

Benjamin Loison

5,7504 gold badges20 silver badges37 bronze badges

answered Nov 13, 2014 at 2:08

kev

163k49 gold badges286 silver badges282 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Zz Tux Over a year ago

JSON does not support circular object

ggorlen Over a year ago

Links to demjson and jsonnet are dead

Drubio Over a year ago

demjson is giving problems with Python 3 because Setuptools has removed support for 2to3. So it may not be a valid alternative for those using python 3.X right now.

hafiz031 Over a year ago

For jsonnet I am getting:

/home/hafiz031/anaconda3/envs/py38/lib/python3.8/site-packages/_jsonnet.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZSt28__throw_bad_array_new_lengthv

Sardar Faisal Over a year ago

ModuleNotFoundError: No module named '_jsonnet'

|

bikram · Accepted Answer · 2021-01-13 08:39:05Z

17

Use json5

import json5

js_obj = '{x:1, y:2, z:3}'

py_obj = json5.loads(js_obj)

print(py_obj)

# output
# {'x': 1, 'y': 2, 'z': 3}

edited Jan 13, 2021 at 8:39

answered Jan 9, 2021 at 18:08

bikram

8,0652 gold badges61 silver badges66 bronze badges

2 Comments

Ice Bear Over a year ago

This is the best one :)

cglacet Over a year ago

Caution: unless you have very small object, don't use JSON5, it's explicitely stated in their documentation that it is slow. And they are not lying, it is very very slow even on average size JSON. Test it on a real usecase befeore adopting this. (I tested version 0.9.8)

Lyhokia · Accepted Answer · 2020-12-24 15:05:44Z

9

I'm facing the same problem this afternoon, and I finally found a quite good solution. That is JSON5.

The syntax of JSON5 is more similar to native JavaScript, so it can help you parse non-standard JSON objects.

You might want to check pyjson5 out.

answered Dec 24, 2020 at 15:05

Lyhokia

1331 silver badge5 bronze badges

2 Comments

Inventor Over a year ago

This is the fastest library. I tried using demjson, my script ran 10.5s. Pyjson5 completes this task in 0.004s.

BigJoe714 Over a year ago

@Inventor can you post an example of how this works?

Chris Billington · Accepted Answer · 2020-03-22 20:38:19Z

4

If you have node available on the system, you can ask it to evaluate the javascript expression for you, and print the stringified result. The resulting JSON can then be fed to json.loads:

def evaluate_javascript(s):
    """Evaluate and stringify a javascript expression in node.js, and convert the
    resulting JSON to a Python object"""
    node = Popen(['node', '-'], stdin=PIPE, stdout=PIPE)
    stdout, _ = node.communicate(f'console.log(JSON.stringify({s}))'.encode('utf8'))
    return json.loads(stdout.decode('utf8'))

answered Mar 22, 2020 at 20:38

Chris Billington

8951 gold badge9 silver badges14 bronze badges

1 Comment

vaduc Over a year ago

After trying other suggestions, I finally finish my problem with this solution. Thank you very much!

chrisb · Accepted Answer · 2014-06-04 01:58:51Z

3

This will likely not work everywhere, but as a start, here's a simple regex that should convert the keys into quoted strings so you can pass into json.loads. Or is this what you're already doing?

In[70] : quote_keys_regex = r'([\{\s,])(\w+)(:)'

In[71] : re.sub(quote_keys_regex, r'\1"\2"\3', js_obj)
Out[71]: '{"x":1, "y":2, "z":3}'

In[72] : js_obj_2 = '{x:1, y:2, z:{k:3,j:2}}'

Int[73]: re.sub(quote_keys_regex, r'\1"\2"\3', js_obj_2)
Out[73]: '{"x":1, "y":2, "z":{"k":3,"j":2}}'

answered Jun 4, 2014 at 1:58

chrisb

52.7k8 gold badges73 silver badges70 bronze badges

Comments

Thiago Lages de Alencar · Accepted Answer · 2020-03-20 14:08:44Z

Not including objects

json.loads()

json.loads() doesn't accept undefined, you have to change to null
json.loads() only accept double quotes
- {"foo": 1, "bar": null}

Use this if you are sure that your javascript code only have double quotes on key names.

import json

json_text = """{"foo": 1, "bar": undefined}"""
json_text = re.sub(r'("\s*:\s*)undefined(\s*[,}])', '\\1null\\2', json_text)

py_obj = json.loads(json_text)

ast.literal_eval()

ast.literal_eval() doesn't accept undefined, you have to change to None
ast.literal_eval() doesn't accept null, you have to change to None
ast.literal_eval() doesn't accept true, you have to change to True
ast.literal_eval() doesn't accept false, you have to change to False
ast.literal_eval() accept single and double quotes
- {"foo": 1, "bar": None} or {'foo': 1, 'bar': None}

import ast

js_obj = """{'foo': 1, 'bar': undefined}"""
js_obj = re.sub(r'([\'\"]\s*:\s*)undefined(\s*[,}])', '\\1None\\2', js_obj)
js_obj = re.sub(r'([\'\"]\s*:\s*)null(\s*[,}])', '\\1None\\2', js_obj)
js_obj = re.sub(r'([\'\"]\s*:\s*)NaN(\s*[,}])', '\\1None\\2', js_obj)
js_obj = re.sub(r'([\'\"]\s*:\s*)true(\s*[,}])', '\\1True\\2', js_obj)
js_obj = re.sub(r'([\'\"]\s*:\s*)false(\s*[,}])', '\\1False\\2', js_obj)

py_obj = ast.literal_eval(js_obj)

Minty · Accepted Answer · 2024-05-18 09:10:52Z

2

Some answers here are outdated, so here's a speed comparison between json5, hjson, and chompjs. (ast.literal_eval and json.loads failed). Evaluated functions on a 1 MB js object to get a good sample. All 3 successes produced an identical dictionary.

#   100.00% - reference time
chompjs.parse_js_object(text)

#   666.65% - 7 times slower
hjson.loads(text)

# 60460.57% - 605 times slower
json5.loads(text)

# fail
ast.literal_eval(text)
json.loads(text)

# won't install on Python 3.11.9
demjson
jsonnet

answered May 18, 2024 at 9:10

Minty

3303 silver badges9 bronze badges

Collectives™ on Stack Overflow

How to convert raw javascript object to a dictionary?

7 Answers 7

`demjson.decode()`

`chompjs.parse_js_object()`

`jsonnet.evaluate_snippet()`

`ast.literal_eval()`

6 Comments

Use json5

2 Comments

2 Comments

1 Comment

Comments

json.loads()

ast.literal_eval()

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

6 Comments

Use json5

2 Comments

2 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related