I am trying to save a large JSON file into a variable using requests module but only part of the JSON is making it into the variable when using the following:
r = requests.get(url)
r.json()
I see there are ways to save it in chunks when writing to a file is there a way to do this when writing to a variable?
response = requests.get(url, stream=True)
with open(path, 'wb') as f:
for chunk in response.iter_content(chunk_size=1024):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
response.close()
Or you can use context manager:
with requests.get(url, stream=True) as response:
with open(path, 'wb') as f:
for chunk in response.iter_content(chunk_size=1024):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
To a variable, it is the same, just use data += chunk
