Pulling paginated json data with python / loop

Question

I pulled json data with api. Response example as below.

{
  "page": 1,
  "pages": 5,
  "limit": 100,
  "count": 466,
  "records": [
    {
      "epoch": "9b4WLoXXYgga9mGRLGsWKu",
      "last_height": 5402,
      "last_time": "2021-02-22T01:02:43.011692Z",
      "expected_blocks": 432,
      "produced_blocks": 432,
    }
  ]
}

But page is limited with 100 rows. Parameters query is available for pages. Below is python code I use:

import requests
import json
import pandas as pd

response = requests.get(url, params = {'page': 1})
json_data = json.loads(response.text)
df = pd.DataFrame(json_data['records']) 
print(df)

How can I get all pages with looping?

Thanks

Divya Prakash · Accepted Answer · 2022-07-25 14:20:57Z

1

You can try this and get all your data in dataframe by looping.

import requests
import json
import pandas as pd

response = requests.get(url, params = {'page': 1}) #---> remember to keep your url 
records = []
for page_number in range(2, response.json().get("pages")+1):
    records += response.json().get('records')    
    response = requests.get(url, params = {'page': page_number})
df = pd.DataFrame(records) 
print(df)

edited Jul 25, 2022 at 14:20

answered Jul 23, 2022 at 21:47

Divya Prakash

1,0141 gold badge7 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Alper Over a year ago

json_data is not defined in your solution. But when I print "records", it didn't work out as expected.

Divya Prakash Over a year ago

oh, sorry, try now.

Alper Over a year ago

@Divya_Prakash I didn't pay attention that first page of the 'records' wasn't print in df. I have changed for page_number in range(1 , ......), so now, it includes first page also. Maybe you want to edit the solution.

Divya Prakash Over a year ago

thanks for flagging this up, I have updated the solution.

Collectives™ on Stack Overflow

Pulling paginated json data with python / loop

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related