Iterate JSON in Python

Question

I am attempting to pull data from an API that returns data as a JSON object. I have the API call which returns data that looks like this (shortened here for brevity's sake)


   "status":"OK",
   "copyright":"Copyright (c) 2017 Pro Publica Inc. All Rights Reserved.",
   "results":[
      {
         "num_results": 10,
         "offset": 0,
         "bills": [
              {
                 "bill_id": "hr2739-113",
                 "bill_type": "hr",
                 "number": "H.R.2739",
                 "bill_uri": "https://api.propublica.org/congress/v1/113/bills/hr2739.json",
                 "title": "Efficient Use of Government Spectrum Act of 2013",
                 "sponsor_title": "Rep.",
                 "sponsor_id": "M001163",
                 "sponsor_name": "Doris Matsui",
                 "sponsor_state": "CA",
                 "sponsor_party": "D",
                 "sponsor_uri": "https://api.propublica.org/congress/v1/members/M001163.json",
                 "gpo_pdf_uri": "http://www.gpo.gov/fdsys/pkg/BILLS-113hr2739ih/pdf/BILLS-113hr2739ih.pdf",
                 "congressdotgov_url": "https://www.congress.gov/bill/113th-congress/house-bill/2739",
                 "govtrack_url": "https://www.govtrack.us/congress/bills/113/hr2739",
                 "introduced_date": "2013-07-18",
                 "committees": "House Armed Services Committee",
                 "committee_codes": ["HSAS","HSIF"],
                 "subcommittee_codes": ["HSAS26","HSIF16"],
                 "primary_subject": "Science, Technology, Communications",
                 "summary_short": "Efficient Use of Government Spectrum Act of 2013 - Directs the Federal Communications Commission (FCC), within three years after enactment of the Middle Class Tax Relief and Job Creation Act of 2012, to: (1) reallocate electromagnetic spectrum between the frequencies from 1755 to 1780 megahertz (currently, such frequencies are occupied by the Department of Defense [DOD] and other federal agencies); and (2) as part of the competitive bidding auctions required by such Act, grant new initial lic...",
                 "latest_major_action_date": "2013-08-29",
                 "latest_major_action": "Referred to the Subcommittee on Intelligence, Emerging Threats &amp; Capabilities."
              },
                           {
                 "bill_id": "hr3355-113",
                 "bill_type": "hr",
                 "number": "H.R.3355",
                 "bill_uri": "https://api.propublica.org/congress/v1/113/bills/hr3355.json",
                 "title": "Reducing Employer Burdens, Unleashing Innovation, and Labor Development Act of 2013",
                 "sponsor_title": "Rep.",
                 "sponsor_id": "G000558",
                 "sponsor_name": "Brett Guthrie",
                 "sponsor_state": "KY",
                 "sponsor_party": "R",
                 "sponsor_uri": "https://api.propublica.org/congress/v1/members/G000558.json",
                 "gpo_pdf_uri": "http://www.gpo.gov/fdsys/pkg/BILLS-113hr3355ih/pdf/BILLS-113hr3355ih.pdf",
                 "congressdotgov_url": "https://www.congress.gov/bill/113th-congress/house-bill/3355",
                 "govtrack_url": "https://www.govtrack.us/congress/bills/113/hr3355",
                 "introduced_date": "2013-10-28",
                 "committees": "House Armed Services Committee",
                 "primary_subject": "Economics and Public Finance",
                 "summary_short": "Reducing Employer Burdens, Unleashing Innovation, and Labor Development Act of 2013 - Expresses the sense of Congress that increasing the competitiveness of U.S. manufacturers will strengthen the national economy. Title I: Investing in America&#39;s Workforce - Investing in America&#39;s Workforce Act - Amends the Workforce Investment Act of 1998 to require state or local workforce investment systems to use youth activities funds allocated to a local area for programs that provide training, which may...",
                 "latest_major_action_date": "2014-01-24",
                 "latest_major_action": "Referred to the Subcommittee on Intelligence, Emerging Threats &amp; Capabilities."
              },

Using Python, I am trying to loop through the data and return all the values like so:

import requests
import json

r = requests.get({url and credentials here}).text

resp = json.loads(r)


for bill in resp['results']['bills']:
    name = bill['results']['bills']['title']
    type = item['results']['bills']['bill_type']

print(name, type)

However, whenever I try to run this, I get

TypeError: list indices must be integers or slices, not str

Why can't I use the list indices as str? There exists plenty of examples that use str.

A dictionary can be accessed with strings as keys, but a list must use integers or slices (e.g.: some_list[3:-1]). — Thomas
– Thomas, Commented Oct 24, 2019 at 5:28

elembie · Accepted Answer · 2019-10-25 03:16:14Z

4

The 'bills' in the json is a list of objects (e.g. "bills" : [ - the opening bracket is square which implies a list).

You have to use an integer to access an element of a list, so you can say;

resp['results'][0]['bills'][0]

To access the first bill, for example.

However, your code is a bit confused in your loop, the bill variable will contain the information for each bill, so you can reference the keys directly, e.g. to loop over the bills in the first result

for bill in resp['results'][0]['bills']:
    name = bill['title']
    type = bill['bill_type']

    print(name, type)

The variables 'name' and 'type' will only contain the details of each bill during the loop block.

You can nest loops to loop through all results

for result in resp['results']:
  for bill in result['bills']:
      name = bill['title']
      type = bill['bill_type']

      print(name, type)

edited Oct 25, 2019 at 3:16

answered Oct 24, 2019 at 5:30

elembie

6403 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Hayden Over a year ago

Perfect! The only edit I had to make was add a colon to the for for in the second example on how to nest loops.

Tanvir Sajed · Accepted Answer · 2019-10-24 05:31:03Z

0

bill is a json object that does not have results or bills as a string index. Try the following :

for bill in resp['results']['bills']:
    name = bill['title']
    type = item['bill_type']

    print(name, type)

answered Oct 24, 2019 at 5:31

Tanvir Sajed

11 bronze badge

1 Comment

Thomas Over a year ago

It's not really a JSON object, it's a Python dictionary which acts much the same as a JSON object. loads deserialises the JSON string into a Python dictionary.

abhilb · Accepted Answer · 2019-10-24 05:34:34Z

0

In json data results is a list of dicts. So first iterate over results. Then for each result iterate over bills to retrieve the name and type of bill.

for result in resp['results']:
    for bill in result['bills']:
        name = bill['title']
        type = item['bill_type']
        print(name, type)

answered Oct 24, 2019 at 5:34

abhilb

5,7672 gold badges22 silver badges26 bronze badges

Comments

lanartri · Accepted Answer · 2019-10-24 05:36:54Z

0

also in your code, name equals:

name = bill['results']['bills']['title'] -> resp['results']['bills']['results']['bills']['title']

answered Oct 24, 2019 at 5:36

lanartri

414 bronze badges

Comments

Shakeel · Accepted Answer · 2019-10-24 17:26:59Z

0

If you are receiving JSON response please find below two methods of parsing it.

Method 1: Normal way - this is something which you are trying

import requests


resp = requests.get(url="url",auth="cred").json()

for bill in resp['results'][0]['bills']:
    bill_name = bill['title']
    bill_type = bill['bill_type']

print(bill_name, bill_type)

Method 2 : Cleaner approach of accessing json response

for bill_dict in resp.get("results", []):  # returns None if "results" field is not found
    for bill in bill_dict.get('bills', []): 
        bill_name = bill.get('title')  # None is assigned to bill_name if key 'title' is not found
        bill_type = bill.get('bill_type')  # None is assigned to bill_name if key 'title' is not found error OR you can assign default value if key is not found Ex: bill.get('bill_type', 'bill type unavailable')

print(bill_name, bill_type)

There are many other ways too.

edited Oct 24, 2019 at 17:26

answered Oct 24, 2019 at 6:59

Shakeel

2,04518 silver badges26 bronze badges

2 Comments

Eduard Ilyasov Over a year ago

For method 2: should be resp.get("results", []) and bill_dict.get('bills', []) because if there's no results or bills you will get None but NoneType object is not iterable.

Shakeel Over a year ago

You are absolutely correct I had added in my code comment but somehow missed to add that point here, thank you for pointing out.

Collectives™ on Stack Overflow

Iterate JSON in Python

5 Answers 5

1 Comment

1 Comment

Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

1 Comment

1 Comment

Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related