Python :ValueError: No JSON object could be decoded

Question

I am accessing a json data and want to convert it in pandas dataframe. Unfortunately, an error occurred when json.loads(req.text)

ValueError: No JSON object could be decoded

Below is my code.

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36",
          "Origin": "https://www.idx.co.id"}
req = requests.get("https://www.idx.co.id/Portals/0/StaticData/HomeHtml/data.js",
                  headers=HEADERS)
stocks = json.loads(req.text)
columns = ['code', 'name']

df = pd.DataFrame([{k: v for k,v in d.items() if k in columns}
              for d in stocks, columns = columns)

Post req.text please

Wonka
– Wonka

2020-03-10 08:59:19 +00:00
Commented Mar 10, 2020 at 8:59 — Wonka
– Wonka, Commented Mar 10, 2020 at 8:59
JavaScript is not JSON.

tfw
– tfw

2020-03-10 09:02:41 +00:00
Commented Mar 10, 2020 at 9:02 — tfw
– tfw, Commented Mar 10, 2020 at 9:02

ionpoint · Accepted Answer · 2020-03-10 09:03:42Z

1

You are not actually receiving a JSON, but a Javascript file. Applying a simple regular expression matching all the data between [] you can achieve the desired result.

import requests
import json
import re

req = requests.get("https://www.idx.co.id/Portals/0/StaticData/HomeHtml/data.js")
content = re.findall(r"= (\[.*?\]);", req.text)
data = json.loads(content[0])
print(data)

Edit: an useful website to test python regexp is https://pythex.org/

answered Mar 10, 2020 at 9:03

ionpoint

8816 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

bkcollection Over a year ago

Thanks. Can you further explain re.findall(r"= (\[.*?\]);", req.text) and the difference json and javascript?

ionpoint Over a year ago

Sure. It searches all data after an = and inside parenthesis before ;. For example applied to this_variable = [{"somedata": 1}]; the result will be [{"somedata": 1}] (which is JSON serializable). For further details and to test it against different strings see the website I added to my answer.

ionpoint Over a year ago

JavaScript is a programming language. JSON is JavaScript Object Notation, which is a data format. en.wikipedia.org/wiki/JSON

bkcollection Over a year ago

I want to understand your first sentence where it is not JSON but javascript file. Can you differentiate how json and javascript in term of format?

Collectives™ on Stack Overflow

Python :ValueError: No JSON object could be decoded

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related