212

Sometimes there is some non-critical asynchronous operation that needs to happen but I don't want to wait for it to complete. In Tornado's coroutine implementation you can "fire & forget" an asynchronous function by simply ommitting the yield key-word.

I've been trying to figure out how to "fire & forget" with the new async/await syntax released in Python 3.5. E.g., a simplified code snippet:

async def async_foo():
    print("Do some stuff asynchronously here...")

def bar():
    async_foo()  # fire and forget "async_foo()"

bar()

What happens though is that bar() never executes and instead we get a runtime warning:

RuntimeWarning: coroutine 'async_foo' was never awaited
  async_foo()  # fire and forget "async_foo()"
Improve this question
3
  • 3
    Related? stackoverflow.com/q/32808893/1639625 In fact, I think it's a duplicate, but I don't want to instant-dupe-hammer it. Can someone confirm? Commented May 20, 2016 at 10:00
  • 8
    @tobias_k, I don't think it's duplicate. Answer at the link is too broad to be answer for this question. Commented May 20, 2016 at 11:44
  • 3
    Does (1) your "main" process continue running forever ? Or (2) do you want to allow your process to die but allowing forgotten tasks continue their job ? Or (3) do you prefer your main process waiting for forgotten tasks just before ending ? Commented May 26, 2016 at 8:23

6 Answers 6

Reset to default
319
+50

Upd:

Replace asyncio.ensure_future with asyncio.create_task everywhere if you're using Python >= 3.7 It's a newer, nicer way to spawn tasks.

asyncio.Task to "fire and forget"

According to python docs for asyncio.Task it is possible to start some coroutine to execute "in the background". The task created by asyncio.ensure_future won't block the execution (therefore the function will return immediately!). This looks like a way to "fire and forget" as you requested.

import asyncio


async def async_foo():
    print("async_foo started")
    await asyncio.sleep(1)
    print("async_foo done")


async def main():
    asyncio.ensure_future(async_foo())  # fire and forget async_foo()

    # btw, you can also create tasks inside non-async funcs

    print('Do some actions 1')
    await asyncio.sleep(1)
    print('Do some actions 2')
    await asyncio.sleep(1)
    print('Do some actions 3')


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Output:

Do some actions 1
async_foo started
Do some actions 2
async_foo done
Do some actions 3

What if tasks are executing after the event loop has completed?

Note that asyncio expects tasks to be completed at the moment the event loop completes. So if you'll change main() to:

async def main():
    asyncio.ensure_future(async_foo())  # fire and forget

    print('Do some actions 1')
    await asyncio.sleep(0.1)
    print('Do some actions 2')

You'll get this warning after the program finished:

Task was destroyed but it is pending!
task: <Task pending coro=<async_foo() running at [...]

To prevent that you can just await all pending tasks after the event loop has completed:

async def main():
    asyncio.ensure_future(async_foo())  # fire and forget

    print('Do some actions 1')
    await asyncio.sleep(0.1)
    print('Do some actions 2')


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    
    # Let's also finish all running tasks:
    pending = asyncio.Task.all_tasks()
    loop.run_until_complete(asyncio.gather(*pending))

Kill tasks instead of awaiting them

Sometimes you don't want to await tasks to be done (for example, some tasks may be created to run forever). In that case, you can just cancel() them instead of awaiting them:

import asyncio
from contextlib import suppress


async def echo_forever():
    while True:
        print("echo")
        await asyncio.sleep(1)


async def main():
    asyncio.ensure_future(echo_forever())  # fire and forget

    print('Do some actions 1')
    await asyncio.sleep(1)
    print('Do some actions 2')
    await asyncio.sleep(1)
    print('Do some actions 3')


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

    # Let's also cancel all running tasks:
    pending = asyncio.Task.all_tasks()
    for task in pending:
        task.cancel()
        # Now we should await task to execute it's cancellation.
        # Cancelled task raises asyncio.CancelledError that we can suppress:
        with suppress(asyncio.CancelledError):
            loop.run_until_complete(task)

Output:

Do some actions 1
echo
Do some actions 2
echo
Do some actions 3
echo
Improve this answer
Sign up to request clarification or add additional context in comments.

16 Comments

@GilAllen this syntax works only in Python 3.5+. Python 3.4 needs old syntax (see docs.python.org/3.4/library/asyncio-task.html ). Python 3.3 and below doesn't support asyncio at all.
@Sardathrion I'm not sure if task points somewhere on thread in which it was created, but nothing stops you from to track them manually: for example, just add all tasks created in thread to a list and when time comes cancel them way explained above.
Sadly this does not work… No idea why. ☹
Great answer. Just would like to point out that it is most of the time not desirable to either wait for all pending tasks to finish OR to cancel all pending tasks. I think there need to be a middle way, ideally collecting the tasks which you want to keep executing after closing the event loop in a "registry" (list, set or something else) and then simply wait for those tasks to finish and cancel all the other ones.
Note that "Task.all_tasks() is deprecated since Python 3.7, use asyncio.all_tasks() instead"
|
35

Output:

>>> Hello
>>> foo() started
>>> I didn't wait for foo()
>>> foo() completed

Here is the simple decorator function which pushes the execution to background and line of control moves to next line of the code.

The primary advantage is, you don't have to declare the function as await

import asyncio
import time

def fire_and_forget(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, *kwargs)

    return wrapped

@fire_and_forget
def foo():
    print("foo() started")
    time.sleep(1)
    print("foo() completed")

print("Hello")
foo()
print("I didn't wait for foo()")

Note: Check my other answer which does the same using plain thread without asyncio.

Improve this answer

6 Comments

I experienced substantial slowdown after using this approach creating ~5 small fire-and-forget tasks per second. Don't use this in production for a long-running task. It'll eat your CPU and memory!
Good work with Django. Don't need Celery, etc. I use it to quickly return the server response to the client's request and then perform the remaining necessary actions that the server response does not depend on. Including actions with Django ORM, as in the usual stream of execution.
Note that this only works from the main thread; asyncio.get_event_loop() raises a RuntimeError if you try this on another thread (Python 3.6 and 3.9 at least). eg threading.Thread(target=lambda: asyncio.get_event_loop()).start() to test.
You wouldn't need to run this another thread. Declare the decorator in the main thread and use the decorator anywhere you would like.
runs lots and lots of threads, not good
|
13

This is not entirely asynchronous execution, but maybe run_in_executor() is suitable for you.

def fire_and_forget(task, *args, **kwargs):
    loop = asyncio.get_event_loop()
    if callable(task):
        return loop.run_in_executor(None, task, *args, **kwargs)
    else:    
        raise TypeError('Task must be a callable')

def foo():
    #asynchronous stuff here


fire_and_forget(foo)
Improve this answer

3 Comments

Nice concise answer. It is worth noting that the executor will default to calling concurrent.futures.ThreadPoolExecutor.submit(). I mention because creating threads is not free; fire-and-forgetting 1000 times a second will probably put a big strain on thread management
Yep. I didn't heed your warning and experienced substantial slowdown after using this approach creating ~5 small fire-and-forget tasks per second. Don't use this in production for a long-running task. It'll eat your CPU and memory!
Is using Process executor better in this case then? @BradSolomon
12

For some reason if you are unable to use asyncio then here is the implementation using plain threads. Check my other answers and Sergey's answer too.

import threading, time

def fire_and_forget(f):
    def wrapped():
        threading.Thread(target=f).start()

    return wrapped

@fire_and_forget
def foo():
    print("foo() started")
    time.sleep(1)
    print("foo() completed")

print("Hello")
foo()
print("I didn't wait for foo()")

produces

>>> Hello
>>> foo() started
>>> I didn't wait for foo()
>>> foo() completed
Improve this answer

3 Comments

If we only need this fire_and_forget functionality and nothing else from asyncio, would it still be better to use asyncio? What's the benefits?
@pir benefit is that you can have an async function in a web server for example, and have a task that you wanna run in the background but return the function asap. For example API route that connects to a DB (async client) then makes a call then closes that connection with DB (async close). I don't really want to await the DB connection close, I just want it to close eventually. I prefer to return the API call asap. This is a perfect use case for asyncio.ensure_future(client.close()) for example. Fire and forget it and return the API response.
The problem here is you have no guarantee that the threads won't get out of hand. With asyncio, they just get 'scheduled', but the main thread will eventually process the foo function. This anser is spawning thread's haphazardly every time the function is called and has no way to catch problems. This may seem similar to @nehem 's other answer, but it functionally is very different under the hood.
1

There's an unspecified termination issue, since "fire and forget" doesn't say when activities must be complete (and ultimately whether to hang or kill them at program termination). The solution is to use a context manager. Python 3.11 now has this as TaskGroup.

An earlier alternative is the aiowire package. Its context manager has a timeout option, and uses a "trampoline" design, allowing async functions to return async functions. This avoids both background threads and infinite async call chains.

Improve this answer

Comments

-4 
def fire_and_forget(f):
    def wrapped(*args, **kwargs):
        threading.Thread(target=functools.partial(f, *args, **kwargs)).start()

    return wrapped

is the better version of the above -- does not use asyncio

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.