Getting pagination data from an API response using python request library

Question

Hi guys I am a python beginner I am trying to fetch all the information from an API whose responses are paginated.I am trying to get the "pagination data" so that I can use that to create my loop using request library.How can write the following shell command in python for me to get the desired results

curl -IXGET https://api.myapplication.com/products?page=1 -H "Content-type: application/json" -H "Authorization: Bearer <PRIVILEGED_ACCESS_TOKEN>" | grep X-Pagination

X-Pagination: {"total_records":3103,"total_pages":32,"first_page":true,"last_page":false,"out_of_bounds":false,"offset":0}

I tried doing this but am getting an error: AttributeError: 'Response' object has no attribute 'type'

import requests
import json
from urllib.request import urlopen,Request

my_headers = {'Authorization': 'Bearer 9YU5OXYdQlBXfROYAXrthdgqli8rkXMAnB5jEmkPZzE'}
pag_results = requests.get('https://api.myapplication.com/products?page=1,headers={"Content-type: application/json"}',headers=my_headers)
pagination_res1 = urlopen(pag_results)
pagination_res2 = pagination_res1.read()
print(pagination_res2)

Nasir Hussain · Accepted Answer · 2021-03-10 11:56:11Z

2

import requests

my_headers = {
    'Authorization': 'Bearer 9YU5OXYdQlBXfROYAXrthdgqli8rkXMAnB5jEmkPZzE',
    'Content-type': 'application/json'
}

page_num = 1
has_next = True

while has_next:
    response = requests.get(
        'https://api.myapplication.com/products?page=' + str(page_num),
        headers=my_headers
    ).json()
    has_next = page_num < response["X-Pagination"]["total_pages"]
    page_num += 1

You don't need to use both urllib and requests. Here is what you need to do:

Keep a counter i.e. page = 0
Do a request to get data
increase page by one
if the page is less than the total pages from the response then do the request again

answered Mar 10, 2021 at 11:56

Nasir Hussain

1051 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

oyugi.collins Over a year ago

Thanks so much @nasir .Do you know why am getting error KeyError: 'X-Pagination' yet when print the header with print(response.headers) I can actually see the key exists

oyugi.collins Over a year ago

the print(response.headers) prints the key after removing the .json()

Nasir Hussain Over a year ago

response["X-Pagination"]["total_pages"] should be based on your response,try printing the response first.

Collectives™ on Stack Overflow

Getting pagination data from an API response using python request library

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related