Running coroutines in different thread with same event loop

Question

I want to run a coroutine in a different thread and get the result that the coroutine returns.

class Main:
    def __init__(self, result_from_io_task=None):
        self._io_task_result = result_from_io_task        

    async def io_task(self):
        await asyncio.sleep(2)
        return "slept of 2s"
    
    def non_async_func(self):
        #This can't be made async.
        if not self._io_task_result:
            #run io_task and get its result

            #event loop will be running in the main thread so I can fire the task
            task = asyncio.create_task(self.io_task)

            #can't await task since I am in non-async func and I cannot 
            #return from non_async_func until and unless I know what
            #self.io_task has returned. Tried following but my app hangs forever.
        
            while not task.done():
                pass

I also tried, but it doesn't work "

def run_in_thread(coro, loop):
       output = []
       def run():
           fut = asyncio.run_coroutine_threadsafe(coro, loop)
           output.append(fut)
       thr = Thread(target=run)
       thr.start()
       return output

async def main():
    main_obj = Main(result_from_io_task=None)
    v = main_obj.non_async_func()

How can I spawn a new thread and run the given coroutine using event loop running in main thread

Does this answer your question? How to call a async function from a synchronized code Python — larsks
– larsks, Commented Aug 10, 2022 at 3:49
What is the reason to run a coroutine in a thread? Both are blocked from loops, and both are effective if you run I/O tasks. — Artyom Vancyan
– Artyom Vancyan, Commented Aug 10, 2022 at 6:25
I don't necessarily need a thread. I have updated my example to show that I start asyncio.run(main()) and within main I want to call a method which may want to get result from an async func. I CANNOT MAKE non_async_func async — user6037143
– user6037143, Commented Aug 10, 2022 at 9:35
Unfortunately, my codebase depends on python <3.8 and asyncio.to_thread is not available in python 3.7. — user6037143
– user6037143, Commented Aug 10, 2022 at 12:19

Artyom Vancyan · Accepted Answer · 2022-08-10 14:05:31Z

1

Unfortunately, my codebase depends on python < 3.8 and asyncio.to_thread is not available in python 3.7

Based on the example of my answer, I'm introducing another implementation of the asynchronous decorator that does not use asyncio.to_thread() function but uses ThreadPoolExecutor instead.

import asyncio
import requests
import concurrent.futures


def asynchronous(func):
    async def wrapper(*args, **kwargs):
        with concurrent.futures.ThreadPoolExecutor() as executor:
            future = executor.submit(func, *args, **kwargs)
            return future.result()

    return wrapper


@asynchronous
def request(url):
    with requests.Session() as session:
        response = session.get(url)
        try:
            return response.json()
        except requests.JSONDecodeError:
            return response.text


async def main():
    task = asyncio.create_task(request("https://google.com/"))
    print("waiting for response...")
    result = await task
    print(result)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()

answered Aug 10, 2022 at 14:05

Artyom Vancyan

5,4023 gold badges17 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

fuzzyTew Over a year ago

This looks like wrapper() will block its event loop until request completes.

Artyom Vancyan Over a year ago

@fuzzyTew, if you call asyncio.create_task(request(...)) multiple times, e.g. in a loop, it will push the request function to the thread pool with the given parameters and wait for the results simultaneously.

fuzzyTew Over a year ago

It looks to me like it waits serially. Any other asyncio tasks in the same thread would block until future.result() returns because it is a threaded future instead of an asyncio future. It works if nothing else is being done in the asyncio loop, but in this case asyncio is not very useful. [Am I wrong?]

Artyom Vancyan Over a year ago

This shows how to run coroutines in multiple threads. As we are dealing with Python threads, this approach will work only for I/O bound tasks to achieve concurrency. If the tasks are CPU bound, then you should probably use processes instead of threads. Ask ChatGPT to explain the difference between concurrency and parallelism, I/O and CPU bound tasks, if you do not know yet. It will help you to solve this kind of problems easily.

fuzzyTew Over a year ago

The approach is okay but it is better to use an asyncio waiting facility on threaded code when the calling function is asyncio, such as asyncio.run_in_executor, simply because it is io-bound and you don’t want to block the other asyncio tasks. Compare with the source code of asyncio.to_thread .

|

Collectives™ on Stack Overflow

Running coroutines in different thread with same event loop

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related