Web Scraping: getting KeyError when parsing JSON in Python

Question

I want to extract the full address from the webpage and I'm using BeautifulSoup and JSON. Here's my code:

import bs4
import json
from bs4 import BeautifulSoup
import requests

url = 'xxxxxxxxxxxxxxxxx'
response = requests.get(url)
html = response.text
soup = BeautifulSoup(html, 'html.parser')

for i in soup.find_all('div', attrs={'data-integration-name':'redux-container'}):
    info = json.loads(i.get('data-payload'))

I printed 'info' out:

{'storeName': None, 'props': {'locations': [{'dirty': False, 'updated_at': '2016-05-05T07:57:19.282Z', 'country_code': 'US', 'company_id': 106906, 'longitude': -74.0001954, 'address': '5 Crosby St  3rd Floor', 'state': 'New York', 'full_address': '5 Crosby St  3rd Floor, New York, 10013, New York, USA', 'country': 'United States', 'id': 17305, 'to_params': 'new-york-us', 'latitude': 40.719753, 'region': '', 'city': 'New York', 'description': '', 'created_at': '2015-01-19T01:32:16.317Z', 'zip_code': '10013', 'hq': True}]}, 'name': 'LocationsMapList'}

What I want is the "full_address" under "location" so my code was:

info = json.loads(i.get('data-payload'))
for i in info['props']['locations']:
        print (i['full_address'])

But I got this error:

----> 5     for i in info['props']['locations']:

KeyError: 'locations'

I want to print the full address out, which is '5 Crosby St 3rd Floor, New York, 10013, New York, USA'.

Thanks a lot!

while iterating your second info doesn't have locations key in 'props' value — dnit13
– dnit13, Commented Sep 16, 2017 at 23:49

PRMoureu · Accepted Answer · 2017-09-17 06:25:05Z

2

The data you are parsing seem to be inconsistent, the keys are not in all objects.

If you still want to perform a loop, you need to use a try/except statement to catch an exception, or the method get to set a fallback when you're looking for a key in a dictionary that could be not here.

info = json.loads(i.get('data-payload'))
for item in info['props'].get('locations', []):
    print (item.get('full_address', 'no address'))

get('locations', []) : returns an empty list if the key location doesn't exist, so the loop doesn't run any iteration.

get('full_address', 'no address') : returns "no adress" in case there is no such key

EDIT :

The data are inconsistent (never trust data). Some JSON objects have a key props with a null /None value. The next fix should correct that :

info = json.loads(i.get('data-payload'))
if info.get('props'):
    for item in info['props'].get('locations', []):
        print (item.get('full_address', 'no address'))

edited Sep 17, 2017 at 6:25

answered Sep 16, 2017 at 23:53

PRMoureu

13.4k6 gold badges46 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Laura Over a year ago

It got another error" ----> 5 for item in info['props'].get('locations', []): AttributeError: 'NoneType' object has no attribute 'get'

Alex Hall · Accepted Answer · 2017-09-16 23:49:52Z

1

Your first object is fine, but it's clear that your second object has no locations key anywhere, nor full_address.

answered Sep 16, 2017 at 23:49

Alex Hall

36.2k5 gold badges63 silver badges98 bronze badges

3 Comments

Laura Over a year ago

'props': {'locations': [{'dirty': False, 'updated_at': '2016-05-05T07:57:19.282Z', 'country_code': 'US', 'company_id': 106906, 'longitude': -74.0001954, 'address': '5 Crosby St 3rd Floor', 'state': 'New York', 'full_address': '5 Crosby St 3rd Floor, New York, 10013, New York, USA', 'country': 'United States', 'id': 17305, 'to_params': 'new-york-us', 'latitude': 40.719753, 'region': '', 'city': 'New York', 'description': '', 'created_at': '2015-01-19T01:32:16.317Z', 'zip_code': '10013', 'hq': True}]}, 'name': 'LocationsMapList'}

Alex Hall Over a year ago

@Laura that's not the second sample object you posted. You can simply Ctrl+F for locations on this page to see that it's in the first object, but not the second.

Laura Over a year ago

Yes, I only pasted several lines from the output. The only part I want is the content of the full address under locations in the first object.

Collectives™ on Stack Overflow

Web Scraping: getting KeyError when parsing JSON in Python

2 Answers 2

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related