Python - Fetching data from API

Question

I have the below code that fetches data from a API. This works fine, however I see that it only returns back 25 rows. I am trying to see how could I extract all of the data from the API call and not limit to 25

import requests
import pandas as pd


API_KEY = API_KEY



url = 'https://api.pagerduty.com/incidents/'
headers = {
    'Accept': 'application/vnd.pagerduty+json;version=2',
    'Authorization': 'Token token={token}'.format(token=API_KEY)}
r = requests.get(url, headers=headers)
data = r.content
data_dict = json.loads(data)
data_df = pd.DataFrame(data_dict['incidents'])

have you tried using r.json() instead od data = r.content -> json.loads(data)? if call was successful, requests will always have the full response (unless the API returned partial data) — py9
– py9, Commented Feb 18, 2020 at 11:37

EugeneProut · Accepted Answer · 2020-02-18 11:38:43Z

3

That is due to pagination in the API response. The call defaults to return 25 incidents. If you pass a limit parameter, you can get more, but only up to 100. If you loop until more is false and increment the offset each call you can get all the incidents.

See here API Reference for limit

answered Feb 18, 2020 at 11:38

EugeneProut

3062 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Kunal K. Kanani · Accepted Answer · 2020-02-18 12:00:57Z

0

Check documentation here.

You can pass the query parameter for example

url = 'https://api.pagerduty.com/incidents?limit=50'

The maximum limit as per docs is 100 so you will have to send multiple GET requests. If there are thousands of records make sure your python script send GET request at specific time interval because of rate-limiting else will through Too Many Requests error.

answered Feb 18, 2020 at 12:00

Kunal K. Kanani

561 silver badge5 bronze badges

Collectives™ on Stack Overflow

Python - Fetching data from API

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related