Write multiple JSON arrays to file

Question

I am retrieving results from an API and have to paginate to get all results. The response is two fields. One is the cursor which I use to paginate, the other is 50 results in an array.

I am successfully loading multiple responses, but am missing a "," between arrays after they are written to the file.

with open(writeFilePath,'a') as outfile:
    while more_results:
      parameters = urllib.parse.urlencode(parms)

      #other code

      if 'cursor' in response_body:
        parms['cursor']=response_body['cursor']
      else: 
        more_results = False

      json.dump(response_body['transactions'],outfile)

Questions:

What is the best way to add a comma between arrays/objects from separate responses?
Should I be writing the comma-separated arrays or drilling down to write the objects in each array? (the latter seems better as I'm writing this)
If yes to #2, how do I dump the contents of response_body['transactions'] rather than the whole array?

I may end up loading 3 yrs of data so instead of continually adding results to one variable and writing it all to the file at the end, I am writing to the file one page at a time.

Loïc Faure-Lacroix · Accepted Answer · 2019-04-25 22:46:21Z

1

This question is difficult to answer because you're trying to solve a problem you may not even have or need to fix.

If you have a lot of data to store, there is no point appending it to the file if you won't be able to load the json later as it will get too big.

If you only need to store the data, you can easily save every request or transaction as individual files.

If you have a lot of data, you should probably just store the data in an actual database instead.

If you really insist in having 1 json file, it probably mean you can load it all at once. Which means that you could simply have a big array in memory and do one single dump at the end of the loading. If you can't do that, there is no point trying to make it in a single file from the start. Which would leave you with using a database or having multiple files (rolling your own database).

answered Apr 25, 2019 at 22:46

Loïc Faure-Lacroix

13.7k7 gold badges72 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Stephen Lloyd Over a year ago

Great counsel, thank you! Ultimately I'm: moving it to a database, hoping to work asynchronously, and maybe going to be parsing out objects into flatter structures. I will write files for each day of data rather than a big file. However, I still need to figure out how to write the JSON to the file in a way that it is readable. I need to dump only the objects, not the whole array. I also need to add commas at the end between pages of the APIs data. I will edit my question.

Stephen Lloyd Over a year ago

yeah, so that took care of everything.

Collectives™ on Stack Overflow

Write multiple JSON arrays to file

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related