python requests fails to decode utf-8 API response

Question

I'm trying to get the response JSON from the following API endpoint https://datos.madrid.es/egob/catalogo/205026-0-cementerios.json. My code is:

import requests

url = 'https://datos.madrid.es/egob/catalogo/205026-0-cementerios.json'
r = requests.get(url)
r.json()

It fails with the error:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

If I get the encoding from the request, it's empty. So I've tried to force the encoding before accesing it, with no success:

import requests

url = 'https://datos.madrid.es/egob/catalogo/205026-0-cementerios.json'
r = requests.get(url)
r.encoding = 'utf-8'
r.json()

gives the same error.

r.text

returns something like:

'\x00\x00\x01\x00\x01\x00  \x00\x00\x01\x00\x18\x0 .......

so looks it's not decoding properly the response.

How can I get it successfully decoded?

Joe Halliwell · Accepted Answer · 2018-12-20 13:36:17Z

4

The server is doing something funky with user agent header (namely returning the favicon if it's not recognised!). You can work around this by forcing the user agent:

url = 'https://datos.madrid.es/egob/catalogo/205026-0-cementerios.json'
r = requests.get(url, headers={"User-Agent": "curl/7.61.0"})
print(r.json())

answered Dec 20, 2018 at 13:36

Joe Halliwell

1,1876 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

koldLight Over a year ago

works perfectly thanks! I didn't realize about the favicon stuff!

grapes · Accepted Answer · 2018-12-20 12:22:33Z

3

It seems to be zipped. Unzip it and then use json.decode. The encoding is utf-8.

Example:

import zlib
decompressed_data=zlib.decompress(f.read(), 16+zlib.MAX_WBITS)

Your URL is public, you can test it with your favorite browser. Chrome gives following headers:

Cache-Control: no-cache
Connection: Keep-Alive
Content-disposition: inline;filename=205026-0-cementerios.json
Content-Encoding: gzip
Content-Length: 4383
Content-Type: application/json;charset=UTF-8
Date: Thu, 20 Dec 2018 12:19:33 GMT
OT-force-Account-Verify: true
Vary: Accept-Encoding
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: IE=8
Xonnection: close

And after unzipping it looks like good json:

{
"@context": {
    "c": "http://www.w3.org/2002/12/cal#",
    "dcterms": "http://purl.org/dc/terms/",
    "geo": "http://www.w3.org/2003/01/geo/wgs84_pos#",
    "loc": "http://purl.org/ctic/infraestructuras/localizacion#",
    "org": "http://purl.org/ctic/infraestructuras/organizacion#",
    "vcard": "http://www.w3.org/2006/vcard/ns#",
    "title": "vcard:fn",
    "id": "dcterms:identifier",
    "relation": "dcterms:relation",
    "references": "dcterms:references",
    "address": "vcard:adr",
    "area": "loc:barrio",
    "district": "loc:distrito",
    "locality": "vcard:locality",
    "postal-code": "vcard:postal-code",
    "street": "vcard:street-address",
    "location": "vcard:geo",
    "latitude": "geo:lat",
    "longitude": "geo:long",
....

answered Dec 20, 2018 at 12:22

grapes

8,7921 gold badge23 silver badges32 bronze badges

1 Comment

koldLight Over a year ago

thanks! I marked the other answer as the accepted one only because it requires less code, but yours works OK also, so I've upvoted it

Collectives™ on Stack Overflow

python requests fails to decode utf-8 API response

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related