GET request with chunks in Python

Question

I'm getting the data from API using requests library doing like this:

import requests

url = "Some String"

headers = {

'Authorization':"Some Token"}

response = requests.request("GET", url, headers=headers)

But the file that I'm trying to get is very large so I receive the time exceptions error. How can I get it using chunks in request?

Thanks!

Possible duplicate of How to download large file in python with requests.py? — RoadRunner
– RoadRunner, Commented Jun 22, 2018 at 12:45
have you seen this SO post? i think using stream=True argument might help. — gyx-hh
– gyx-hh, Commented Jun 22, 2018 at 13:55

Keithx · Accepted Answer · 2018-06-27 11:41:19Z

0

import requests    
import datetime
import pandas as pd


url = "some URL"
headers = {

'Authorization':"Some Token"}

start_date = datetime.datetime(2018, 6, 1)
end_date = datetime.datetime.now()
temp_end_date = start_date + datetime.timedelta(days=7)

output = dict()
while temp_end_date <= end_date:
    temp_url = url % (start_date.timestamp()*1000, temp_end_date.timestamp()*1000)
    response = requests.get(temp_url, headers=headers)
    temp_data = response.json()
    for key, value in temp_data.items():
        output_arr = output.get(key, [])
        output_arr.extend(value)
        output[key] = output_arr
    start_date = temp_end_date + datetime.timedelta(seconds=1)
    temp_end_date += datetime.timedelta(days=7)

data=output
df=pd.DataFrame(data)

df.head()

answered Jun 27, 2018 at 11:41

Keithx

3,15816 gold badges46 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

GET request with chunks in Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related