how to speed up python backend making lots of http requests

Question

I'm trying to learn some network/backend stuff. I now want to build an API that makes an HTTP request, does some processing, sends back a response. Not very useful, but it's for learning.

I noticed that the get request is a huge bottleneck. It is a I/O problem i think because the respones are veery small. Now I thought I could maybe do the downloading on multiple threads. If a fictional client of mine makes a request, an URL would need to be added to a pool, then fetched (by some worker thread) und returned to the worker thread, processed, and send back. Or something like that...

I'm really not an expert and maybe nothing what I just said made any sense... but I would really appreciate a little help:)

You can make a quantum leap using asyncio. It requires a bit of learning but it's worth it. To build an API I recommend aiohttp. Async programming is great to solve the network I/O bottleneck. We have used it in two big scale projects, with excellent results — Pynchia
– Pynchia, Commented May 14, 2020 at 23:22
You didn't react at all to my answer. Did I misinterpret what you wanted to know? — gelonida
– gelonida, Commented May 20, 2020 at 23:28
You answer was perfect. I did it with threading. I'm now looking into asynchronous stuff... — itsame
– itsame, Commented May 21, 2020 at 21:23

gelonida · Accepted Answer · 2020-05-14 23:22:56Z

Multiple solutions exist. You can use threading (thread pools) or multiprocessing (multiprocessing pools) to perform multiple requests in parallel.

Or you could use libraries like asyncio (or twisted) to perform multiple requests within one thread in a way, that waiting for IO is no more the blocking point.

I suggest you look at:

https://docs.python.org/3/library/threading.html for threading or https://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing#module-multiprocessing for multiprocessing.

Asynchronous programming is in my opinion much more difficult, but if curious look at https://docs.python.org/3/library/asyncio.html?highlight=asyncio#module-asyncio for asyncio basics and at https://docs.aiohttp.org/en/stable/ for performing multiple http requests in 'parallel' with asyncio

Afterwards after playing a little you will probably have much pore precise questions.

Just post your code then, explain issues and you will get more help

Collectives™ on Stack Overflow

how to speed up python backend making lots of http requests

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related