Best practice: send multiple files to rest endpoint using python requests

Question

What is the best way to send a lot of POST requests to a REST endpoint via Python? E.g. I want to upload ~500k files to a database.

What I've done so far is a loop that creates for each file a new request using the requests package.

# get list of files
files = [f for f in listdir(folder_name)]
# loop through the list 
for file_name in files:
   try: 
      # open file and get content
      with open(folder_name + "\\" + file_name, "r") as file:
         f = file.read()
         # create request
         req = make_request(url, f)

         # error handling, logging, ...

But as this is quite slow: what is the best practice to do that? Thank you.

[f for f in listdir(folder_name)] is redundant. just use for file_name in listdir(folder_name) — Jab
– Jab, Commented Feb 25, 2019 at 15:34

user_D_A__ · Accepted Answer · 2019-02-25 15:17:31Z

1

First approach:

I dont know if it is the best practice you can split the files in batches of 1000 and zip it and send it as post requests using threads ( set the num threads = number of processor cores)

( The rest end point can extract the zipped contents and then process it )

second approach:

zip the files in batches and transfer it in batches after the transfer is completed , validate in the server side Then start the database upload at one go.

answered Feb 25, 2019 at 15:17

user_D_A__

4602 gold badges6 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ray Over a year ago

Zip cloud help to reduce the amount of files, but still that are many files.But I like the approach with multiple threads of different cores.

MrName · Accepted Answer · 2019-02-25 16:31:36Z

1

The first thing you want to do is determine exactly which part of your script is the bottleneck. You have both disk and network I/O here (reading files and sending HTTP requests, respectively).

Assuming that the HTTP requests are the actual bottleneck (highly likely), consider using aiohttp instead of requests. The docs have some good examples to get you started and there are plenty of "Quick Start" articles out there. This would allow your network requests to be cooperative, meaning that other python code can run while one of your network requests is waiting. Just to be careful to not overwhelm whatever server is receiving the requests.

answered Feb 25, 2019 at 16:31

MrName

2,53918 silver badges32 bronze badges

2 Comments

ray Over a year ago

Thanks for the reading material, but this doesn't block my program. But still is this good (best) practice to upload a lot of files?

MrName Over a year ago

I'm not sure that there is any best practice in regards to how to approach this. In general, async will help speed up situations where you need to make a large number of network requests.

Collectives™ on Stack Overflow

Best practice: send multiple files to rest endpoint using python requests

2 Answers 2

First approach:

second approach:

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

First approach:

second approach:

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related