Shorten this code... Append JSON data from Multiple GET requests, then dump into single Excel File with Headers

Question

I'm very new to Python.

The API I am using, is limited to 1000 results per request. I'm trying to get all the results/data for a specific date range, 1 month for this example.

Before I start, I run a GET request to see how many results would need to be exported in total.

For this example, the total = 14,841. I then create 15 queries/requests of 1000 each, using the "start" parameters of 1001, 2001, 3001, etc, and then take the response for each request and save that into a txt file containing the JSON data.

I am sure there is a way to shorten this code & process using Python. There are times when the Total is over 500,000 results, which forces me to do 500+ requests.

Below is an example of 5 requests / queries.

How can I shorten this code to combine multiple queries & data into 1 file?

import requests

url = "https://api.leadspedia.com/core/v2/leads/getAll.do"

headers = { 'Content-Type': "application/json", 'Authorization': "Basic XYZ" }

querystring = {"fromDate":"2020-04-01","toDate":"2020-04-30","start":"0","limit":"1000"} querystring2 = {"fromDate":"2020-04-01","toDate":"2020-04-30","start":"1001","limit":"1000"} querystring3 = {"fromDate":"2020-04-01","toDate":"2020-04-30","start":"2001","limit":"1000"} querystring4 = {"fromDate":"2020-04-01","toDate":"2020-04-30","start":"3001","limit":"1000"} querystring5 = {"fromDate":"2020-04-01","toDate":"2020-04-30","start":"4001","limit":"1000"}

response = requests.request("GET", url, headers=headers, params=querystring) response2 = requests.request("GET", url, headers=headers, params=querystring2) response3 = requests.request("GET", url, headers=headers, params=querystring3) response4 = requests.request("GET", url, headers=headers, params=querystring4) response5 = requests.request("GET", url, headers=headers, params=querystring5)

file = open("out1.txt", "wb") file.write(response.content) file.close()

file = open("out2.txt", "wb") file.write(response2.content) file.close()

file = open("out3.txt", "wb") file.write(response3.content) file.close()

file = open("out4.txt", "wb") file.write(response4.content) file.close()

file = open("out5.txt", "wb") file.write(response5.content) file.close()

Michał · Accepted Answer · 2022-04-23 15:32:53Z

0

You can create a boolean indicating whether you are still obtaining new, valid results and a while loop using it:

query_number = 1
start = 0
more_to_query = True

while more_to_query:
  querystring={"fromDate":"2020-04-01","toDate":"2020-04-30","start":str(start),"limit":"1000"} 
  response = requests.request("GET", url, headers=headers, params=querystring) 
  file = open("out" + str(query_number) + ".txt", "wb")
  file.write(response.content) 
  file.close()
  start += 1000
  query_number += 1
  # set more_to_query to False when you detect you're no longer receiving new valid responses

answered Apr 23, 2022 at 15:32

Michał

1345 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Keegan Dhue Over a year ago

So if I now wanted to append all those requests into 1 file and export to excel, using the info from the responses as headers? Can that also be done in Python?

Michał Over a year ago

I usually use pandas library to deal with data, it has a function to save to excel (to_excel), you can also set the headers there. pandas.pydata.org/docs/user_guide/10min.html Basically you would have to create a dataframe with requests as rows, I believe.

Collectives™ on Stack Overflow

Shorten this code... Append JSON data from Multiple GET requests, then dump into single Excel File with Headers

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related