Asyncio not running Aiohttp requests in parallel

Question

I want to run many HTTP requests in parallel using python. I tried this module named aiohttp with asyncio.

import aiohttp
import asyncio

async def main():
    async with aiohttp.ClientSession() as session:
        for i in range(10):
            async with session.get('https://httpbin.org/get') as response:
                html = await response.text()
                print('done' + str(i))

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

I expect it to execute all the requests in parallel, but they are executed one by one. Although, I later solved this using threading, but I would like to know what's wrong with this?

Related stackoverflow.com/q/63872924/13782669

alex_noname
– alex_noname

2021-03-20 20:38:57 +00:00
Commented Mar 20, 2021 at 20:38 — alex_noname
– alex_noname, Commented Mar 20, 2021 at 20:38
Related : docs.aiohttp.org/en/stable/client_quickstart.html

Mooncrater
– Mooncrater

2021-11-13 16:12:56 +00:00
Commented Nov 13, 2021 at 16:12 — Mooncrater
– Mooncrater, Commented Nov 13, 2021 at 16:12

Krishna Chaurasia · Accepted Answer · 2021-03-20 15:54:04Z

16

You need to make the requests in a concurrent manner. Currently, you have a single task defined by main() and so the http requests are run in a serial manner for that task.

You could also consider using asyncio.run() if you are using Python version 3.7+ that abstracts out creation of event loop:

import aiohttp
import asyncio

async def getResponse(session, i):
    async with session.get('https://httpbin.org/get') as response:
        html = await response.text()
        print('done' + str(i))

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [getResponse(session, i) for i in range(10)] # create list of tasks
        await asyncio.gather(*tasks) # execute them in concurrent manner

asyncio.run(main())

edited Mar 20, 2021 at 15:54

answered Mar 20, 2021 at 8:21

Krishna Chaurasia

9,6546 gold badges27 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Carlo Zanocco Over a year ago

You can simplify the loop with list comprehension tasks = [asyncio.create_task(getResponse(session, i)) for i in range(10)]

Krishna Chaurasia Over a year ago

thanks for the feedback, list comprehension should definitely be preferred.

Anshit Bhardwaj Over a year ago

Understood! Thank you so much!

dirn Over a year ago

Unless you specifically need the Task associated with each getResponse, you can omit create_task. [getResponse(session, i) for i in range(10)]

Suganthan Raj Over a year ago

I have used the above code.. once all the execution completed. again it executing and showing errors

Collectives™ on Stack Overflow

Asyncio not running Aiohttp requests in parallel

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related