Python - Loop through each page to get all records

Question

I would like to retrieve all records (total 50,000) from an API endpoint. The endpoint only returns a maximum of 1000 records per page. Here's the function to get the records.

def get_products(token,page_number):
  url = "https://example.com/manager/nexus?page={}&limit={}".format(page_number,1000)
  header = {
    "Authorization": "Bearer {}".format(token)
  }
  response = requests.get(url, headers=header)
  product_results = response.json()

  total_list = []

  for result in product_results['Data']:
    date = result['date']
    price = result['price']
    name = result['name']
    total_list.append((date,price,name))
  columns = ['date', 'price', 'name']
  df = pd.DataFrame(total_list, columns=columns)
  results = json.dumps(total_list)
  return df, results

How can I loop through each page until the final record without hardcoding the page numbers? Currently, I'm hardcoding the page numbers as below for the first 2 pages to get 2000 records as a test.

for page_number in np.arange(1,3):
  token = get_token()

  product_df,product_json = get_products(token,page_number)
  if page_number==1:
    product_all=product_df
  else:
    product_all=pd.concat([product_all,product_df])

print(product_all)

Thank you.

frogcoder · Accepted Answer · 2021-08-16 04:29:43Z

1

I don't know about the behavior of the endpoint. Assuming when the page number is greater than the last page number, you would get an empty list instead. If that is the case, you could just check if the result is empty.

page_number = 1
token = get_token()

product_df, product_json = get_products(token,page_number)
product_all=product_df

while product_df.size:
  page_number = page_number + 1
  token = get_token()

  product_df,product_json = get_products(token,page_number)
  product_all=pd.concat([product_all,product_df])

print(product_all)

If you are sure there are 1000 records max per page, you could check if the result count is less than 1000 and stop the loop.

answered Aug 16, 2021 at 4:29

frogcoder

1,0031 gold badge8 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

chanrlc · Accepted Answer · 2021-08-16 04:51:22Z

0

It depends on how your backend method: json GET's return. page and limit are required. you may rewrite the json return all data. in stead of just every 1000.

num = int(50000/1000);  
    
for i in range(1, num):
     token = get_token()
     product_df,product_json = get_products(token, i)
    if i==1:
       product_all=product_df
    else:
       product_all=pd.concat([product_all,product_df])

print(product_all)

edited Aug 16, 2021 at 4:51

answered Aug 16, 2021 at 4:05

chanrlc

1822 silver badges11 bronze badges

2 Comments

Tim Over a year ago

Thanks. I dont have control over the API configuration. Since it is setup to only return 1000 records per page, I think I have to do a loop and that's where my second part comes in. I'm not sure how to loop through each of the pages until the last page.

chanrlc Over a year ago

ok, it needs to calculate page number and concat each result.

Collectives™ on Stack Overflow

Python - Loop through each page to get all records

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related