Python to retrieve multiple pages of data from API with GET

Question

I am trying to use Python 3 requests.get to retrieve the data from this page, using its API. I am interested in retrieving data from all the pages using the API.

Here is my attempt so far

data = 'https://api.safecast.org/en-US/measurements'
data = requests.get(url)

My problem is the following - when I check the length of data using

len(data.json())

it gives me 25. This is because there are 25 records per page and it is only returning page number 1. I need to retrieve data from all pages, not just page 1.

According to the API, there are some parameters that can be specified in the query in order to filter the search. But, I do not know how to specify the page number in the query.

I looked through these 2 SO posts (1, 2), but I could not find something relevant to my problem.

Based on this post, I tried

print(data.links)

but this just gave {}

Question

Is there a way to collect data from all pages at once, using the API? Also, how do I determine the number of pages programmatically?

Amit Nanaware · Accepted Answer · 2018-12-14 05:37:17Z

5

Hi As there are total 4458708 pages , you can add for loop and get json of each page. Check below code

import requests
for page in range(1,4458709):
    url = 'https://api.safecast.org/en-US/measurements.json?page=%s'%page
    data = requests.get(url)
    print data.json()

answered Dec 14, 2018 at 5:37

Amit Nanaware

3,3961 gold badge9 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

edesz Over a year ago

Thanks, but is there a way to automatically determine the number of pages?

Amit Nanaware Over a year ago

you can parse the response of data.content and get the total number of pages using below code url= 'https://api.safecast.org/en-US/measurements' data = requests.get(url) print data.content The content is in the form of html

edesz Over a year ago

Thanks. I'll try to work with this and see what it can give me.

Collectives™ on Stack Overflow

Python to retrieve multiple pages of data from API with GET

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related