How to extract JSON from script with Python?

Question

I am parsing a scraped html page that contains a script with JSON inside. This JSON contains all info I am looking for but I can't figure out how to extract a valid JSON.

Minimal example:

my_string = '
        (function(){
          window.__PRELOADED_STATE__ = window.__PRELOADED_STATE__ || [];
          window.__PRELOADED_STATE__.push(
        
           { *placeholder representing valid JSON inside* }
        );
        })()
'

The json inside is valid according to jsonlinter.

The result should be loaded into a dictionary:

import json
import re
my_json = re.findall(r'.*(?={\").*', my_string)[0] // extract json
data = json.loads(my_json)
// print(data)

regex: https://regex101.com/r/r0OYZ0/1

This try results in:

>>> data = json.loads(my_json)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 7 (char 6)

How can the JSON be extracted and loaded from the string with Python 3.7.x?

You have to use the json.loads method (don't forget the s). But your string doesn't seem to be valid json. — herveRilos
– herveRilos, Commented Dec 20, 2020 at 8:53
The JSON you have in your regex example is not a valid JSON: {"publicRuntimeConfig":{"public } — PApostol
– PApostol, Commented Dec 20, 2020 at 9:17

adir abargil · Accepted Answer · 2020-12-20 11:04:24Z

1

you can try to extract this regex, its a very simple case and might not answerto all possible json variations:

my_string = '''
        (function(){
          window.__PRELOADED_STATE__ = window.__PRELOADED_STATE__ || [];
          window.__PRELOADED_STATE__.push(
        
            {"tst":{"f":3}}
        );
        })()
'''
result = re.findall(r"push\(([{\[].*\:.*[}\]])\)",string3)[0]
result
>>> '{ "tst":{"f":3}}'

to parse it to dictionary now:

import json 

dictionary = json.loads(result)
type(dictionary)
>>>dict

edited Dec 20, 2020 at 11:04

answered Dec 20, 2020 at 9:40

adir abargil

5,7453 gold badges23 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

merlin Over a year ago

Yes, the Json contains escaped characters unfortunatelly, do you know how to strip the escape sign?

merlin Over a year ago

the json contains: : ["string 5\'-mono. D

merlin Over a year ago

I updated the question and linked to regex101 with the full JSON

merlin Over a year ago

This is because the regex is not OK. The lookbehind captures the closing bracked. If you remove the ) at the end it will validat according to jsonlint.com

merlin Over a year ago

You are right! Looks I have another regex that works. .*(?:push()(.*)) Which one would you recommend using and why?

|

PApostol · Accepted Answer · 2020-12-20 09:26:32Z

1

Have a look at the below. Note that { *placeholder representing valid JSON inside* } has to be a valid JSON.

my_string = '''
        <script>
            (function(){
              window.__PRELOADED_STATE__ = window.__PRELOADED_STATE__ || [];
              window.__PRELOADED_STATE__.push(
            
               {"foo":["bar1", "bar2"]}
            );
            })()
         </script>
'''

import re, json

my_json = re.findall(r'.*(?={\").*', my_string)[0].strip()
data = json.loads(my_json)
print(data)

Output:

{'foo': ['bar1', 'bar2']}

answered Dec 20, 2020 at 9:26

PApostol

2,3122 gold badges17 silver badges24 bronze badges

2 Comments

merlin Over a year ago

Getting closer. I have now a similar regex: .*(?:push()(.*)(?<=)). You are right about the valid json. There is a escape char inside the JSON: ["string 5\'-mono. Do you know a function that strips those?

PApostol Over a year ago

Maybe you can just use str.replace()

drumhellerb · Accepted Answer · 2020-12-20 08:48:45Z

0

The my_string provided here is not valid JSON. For valid JSON, you can use json.loads(JSON_STRING)

import json

d = json.loads('{"test":2}')
print(d) # Prints the dictionary `{'test': 2}`

answered Dec 20, 2020 at 8:48

drumhellerb

3811 silver badge7 bronze badges

3 Comments

merlin Over a year ago

Well that is the whole point! Please read my question again, the "json inside is valid". I need to figure out how to remove the javascript code around it.

drumhellerb Over a year ago

The question was edited to clarify that after I posted this answer.

merlin Over a year ago

Yes, after I saw your answer I tried to make things more clear. A regex might help, any ideas?

Max Ehsan · Accepted Answer · 2024-08-26 06:48:04Z

Late answer, but I prefer processing the HTML document with beautifulsoup or selectolax and extracting the relevant script node. React apps tend to have huge amount of json payload, and regex extraction may fail.

Here's a relevant snippet from one of my older projects:

import orjson as json
from selectolax.parser import HTMLParser

def extract_redux_json_from_html_page(html: bytes):
    redux_state = "__REDUX_STATE__" # window.__PRELOADED_STATE__
    parser = HTMLParser(html)
    for script in parser.css("script"):
        code = script.text(strip=True).strip()

        if redux_state not in code:
            continue

        lines = [x.strip() for x in code.split("\n") if redux_state in x]
        if any(lines):
            json_str = lines[0].split("=", 1)[1]
            return json.loads(json_str)

The above code assumes the react/redux JSON payload is in a single line (which usually holds true for nodejs produced minified documents). Modifications may be required for non-uglified code fragments.

Another option would be to run the HTML document through pyv8, chakra or a javascript engine and extract the relevant fragment from inside JS. This is a much slower alternative, but may be necessary in some specific cases; e.g. if the JSON object has been scrambled and needs to execute JS code to restore it to original source.

Here's a snippet that scrapes vuejs/nuxtjs encrypted JSON payload:

import orjson as json
from py_mini_racer.py_mini_racer import MiniRacer
from selectolax.parser import HTMLParser

def descramble_nuxt_json(html: str | bytes):
    dom = HTMLParser(html=html)
    for script in dom.css("script"):
        code = script.text().strip()
        if "window.__NUXT__" in code:
            code = code.replace("window.__NUXT__", "__NUXT__OBJ__")
            code += "\nJSON.stringify(__NUXT__OBJ__);"
            return json.loads(MiniRacer().eval(code))

Collectives™ on Stack Overflow

How to extract JSON from script with Python?

4 Answers 4

7 Comments

2 Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

7 Comments

2 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related