Executing coroutines created using async/await in across multiple process invocations across machines

Question

I understand the basic examples shown with using python 3.5 and its new async/await functionality (which is super cool).

One of the things I am exploring is how I might be able to start some async work, and once some blocking work is started, serialize (pickle?) the state of the coroutine, and then at some deferred point in time (possibly on a different machine), restart the same program, reload the state, and attempt to make progress, and repeat until the async work is done.

Specifically in the diagram at https://docs.python.org/3/library/asyncio-task.html#example-chain-coroutines ... I am wondering if I can replace event loop to accomplish this. How might I go about doing such a thing?

I am hoping someone with more expertise in python's coroutines, asyncio and new async/await functionality maybe able to shed light on how to do this either using the approach I am trying by sharing some pointers, or quite possibly suggest something better.

Andrew Svetlov · Accepted Answer · 2015-11-10 06:39:48Z

1

asyncio library itself has only sockets, pipes and processes.

For complex interprocess communications you need something built on top of asyncio.

Your example describes typical message queue usage, e.g. RabbitMQ with asynqp library.

Or you may build something from scratch, say, using aiozmq.

You question is too broad for making strict answer.

answered Nov 10, 2015 at 6:39

Andrew Svetlov

17.5k8 gold badges70 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Nikhil Kothari Over a year ago

I am not trying to make an async library for X (where X = RabbitMQ etc.) ... asyncio is more than just those. For example, event_loop comes from asyncio, and is responsible for executing coroutines to completion. I want to understand how one might build a custom event loop/executor to introduce ability to serialize coroutines, and then pick up from where they were left off in a future/another instance of the process on some other machine. Hopefully that clarifies as well as make a bit more specific?

Andrew Svetlov Over a year ago

No, using custom event_loop or executor for tasks like you wish is wrong. It should be done by a library using standard event loop. The library might provide future-like objects for tasks executed on another machine, sure.

Nikhil Kothari Over a year ago

Yes, assume there is a library for the tasks themselves to produce future-like objects. However I want the main program that started the task to not sit there blocked on async work consuming resources, but instead to end the process and then pick up at a later point.

Collectives™ on Stack Overflow

Executing coroutines created using async/await in across multiple process invocations across machines

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related