Asyncio RuntimeError: Event Loop is Closed

Question

I'm trying to make a bunch of requests (~1000) using Asyncio and the aiohttp library, but I am running into a problem that I can't find much info on.

When I run this code with 10 urls, it runs just fine. When I run it with 100+ urls, it breaks and gives me RuntimeError: Event loop is closed error.

import asyncio
import aiohttp


@asyncio.coroutine
def get_status(url):
    code = '000'
    try:
        res = yield from asyncio.wait_for(aiohttp.request('GET', url), 4)
        code = res.status
        res.close()
    except Exception as e:
        print(e)
    print(code)


if __name__ == "__main__":
    urls = ['https://google.com/'] * 100
    coros = [asyncio.Task(get_status(url)) for url in urls]
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait(coros))
    loop.close()

The stack trace can be found here.

Any help or insight would be greatly appreciated as I've been banging my head over this for a few hours now. Obviously this would suggest that an event loop has been closed that should still be open, but I don't see how that is possible.

is not Asyncio error. Python recursive error, reached limit. need thread for all non class function... — dsgdfg
– dsgdfg, Commented Sep 16, 2015 at 5:17
First, make sure you are using the latest aiohttp release. I assume you do. Technically aiohttp need one loop iteration after finishing request for closing underlying sockets. So insert loop.run_until_complete(asyncio.sleep(0)) before loop.close() call. — Andrew Svetlov
– Andrew Svetlov, Commented Sep 16, 2015 at 6:00
Your traceback suggests that a job submitted to an Executor through run_in_executor returned after the loop has been closed. Weirdly enough, aiohttp and asyncio don't use run_in_executor... — Vincent
– Vincent, Commented Sep 16, 2015 at 13:11
@AndrewSvetlov, thanks for the reply - I tried sleeping before close, but still no dice... any other ideas? — Patrick Allen
– Patrick Allen, Commented Sep 16, 2015 at 13:38
@Vincent technically they does, DNS resolving is performed by run_in_executor -- but it should be done before finishing get_status tasks. — Andrew Svetlov
– Andrew Svetlov, Commented Sep 16, 2015 at 13:44

Andrew Svetlov · Accepted Answer · 2015-10-26 20:12:24Z

19

The bug is filed as https://github.com/python/asyncio/issues/258 Stay tuned.

As quick workaround I suggest using custom executor, e.g.

loop = asyncio.get_event_loop()
executor = concurrent.futures.ThreadPoolExecutor(5)
loop.set_default_executor(executor)

Before finishing your program please do

executor.shutdown(wait=True)
loop.close()

edited Oct 26, 2015 at 20:12

answered Sep 16, 2015 at 17:43

Andrew Svetlov

17.5k8 gold badges70 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Patrick Allen Over a year ago

Awesome Andrew, thanks for your help. I didn't realize I was talking to part of the team :). Following this on GH

RomanPerekhrest Over a year ago

Changed in version 3.5.3: BaseEventLoop.run_in_executor() no longer configures the max_workers of the thread pool executor it creates

RomanPerekhrest Over a year ago

Andrew, can you suggest not "quick workaround" but some robust workaround for Python 3.5 ?

Vincent · Accepted Answer · 2015-12-24 09:50:25Z

7

You're right, loop.getaddrinfo uses a ThreadPoolExecutor to run socket.getaddrinfo in a thread.

You're using asyncio.wait_for with a timeout, which means res = yield from asyncio.wait_for... will raise a asyncio.TimeoutError after 4 seconds. Then the get_status coroutines return None and the loop stops. If a job finishes after that, it will try to schedule a callback in the event loop and raises an exception since it is already closed.

edited Dec 24, 2015 at 9:50

answered Sep 16, 2015 at 14:26

Vincent

13.5k1 gold badge51 silver badges68 bronze badges

5 Comments

Patrick Allen Over a year ago

Ahh, that makes sense, but this is the only way I have found to implement request timeouts. Do you know of a way that I could timeout without closing the loop?

Vincent Over a year ago

@PatrickAllen You might want to increase the number of workers that is 5 by default.

Vincent Over a year ago

@PatrickAllen Or use loop._default_executor.shutdown(wait=True) before closing the loop.

Patrick Allen Over a year ago

I'll mark this as answered, because this seems to have fixed the original problem. Should I be limiting the max number of connections? It seems that requests are timing out for no apparent reason. Maybe I'm making too many requests too quickly?

Vincent Over a year ago

@PatrickAllen Well, 5 worker threads and a thousand of request means you're trying to run 200 socket.getaddrinfo in 4 seconds which seems reasonable to me, even though the number of workers can be increased. You can also give a custom TcpConnector to request in order to specify a connection timeout: connector=aiohttp.TCPConnector(loop=loop, force_close=True, conn_timeout=1)

Arhadthedev · Accepted Answer · 2022-10-26 18:44:44Z

3

This is a bug in the interpreter. Fortunately, it was finally fixed in 3.10.6 so you just need to update your installed Python.

answered Oct 26, 2022 at 18:44

Arhadthedev

2491 gold badge7 silver badges19 bronze badges

Collectives™ on Stack Overflow

Asyncio RuntimeError: Event Loop is Closed

3 Answers 3

3 Comments

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related