Implement coroutine in terms of async def

Question

The following example defines a coroutine multiply, which waits for a value, multiplies it by a constant factor, and prints the result. Another function, product_table, uses multiply to produce product tables.

def multiply(factor):
    print(f"product table for {factor}")
    while True:
        sent_value = yield
        result = factor * sent_value
        print(f"{factor} x {sent_value} = {result}")

def product_table(coro):
    coro.send(None) # start coroutine
    for value in range(1, 11):
        coro.send(value)

product_table(multiply(3))

running the example produces:

product table for 3
3 x 1 = 3
3 x 2 = 6
3 x 3 = 9
3 x 4 = 12
3 x 5 = 15
3 x 6 = 18
3 x 7 = 21
3 x 8 = 24
3 x 9 = 27
3 x 10 = 30

I am trying to implement the exact same example in terms of async def and await, but I'm getting nowhere. My original, incorrect expectation was that the following coroutine function using await was equivalent to the coroutine relying on yield:

async def multiply(factor):
    print(f"product table for {factor}")
    while True:
        await sent_value
        result = factor * sent_value
        print(f"{factor} x {sent_value} = {result}")

It may sound stupid that I expected this to work, but to my dull brain it reads "wait for the sent value". This is not the case -- I get NameError: name 'sent_value' is not defined.

So far, my limited understanding is that you can define coroutine functions in terms of yield (as in the first, working example), or in terms of async def and await, as I attempted in the second example. This does not seem to be the case.

My concrete questions are:

How would you implement the first example in terms of async def?
Does it even make sense to implement in terms of async def?

I hate that all the freaking examples I've found are in terms of fake pipelines that use time.sleep(0.1) everywhere. I'm trying to make a blog post with examples that use more concrete (if also trivial) pipelines.

[removed erroneous edit]

async def started out as syntactic sugar, yes, to make it easier to spot that you have a coroutine (as def functionname() does not communicate that somewhere in the body there's a yield or yield from). I'm putting this in the comments because that part of your question is tangental to your actual issue and makes your question borderline too broad. I'd take it out. — Martijn Pieters
– Martijn Pieters, Commented Nov 18, 2017 at 17:38
What you implemented in your edit doesn't work: TypeError: object range can't be used in 'await' expression. A range() object is not an awaitable. — Martijn Pieters
– Martijn Pieters, Commented Nov 18, 2017 at 17:57
Concrete example: build a web service that aggregates information from a bunch of different sources, then ranks those in a list. Say you have a bunch of REST APIs to call, some databases, some file data to process, then aggregate all that and run some rankers on those, then finally sort the results. That's an excellent use-case for awaitables, because you can wait for data to appear in parallel. That's basically how the Facebook and Instagram news feeds work.. — Martijn Pieters
– Martijn Pieters, Commented Nov 18, 2017 at 18:07
See github.com/Instagram/django-workload/blob/master/… for a super-simplified implementation of such a setup. — Martijn Pieters
– Martijn Pieters, Commented Nov 18, 2017 at 18:08

Mikhail Gerasimov · Accepted Answer · 2017-11-20 05:09:25Z

3

Does it even make sense to implement in terms of async def?

No, I think it's not.

It happened that asyncio module was implemented using generators, but, theoretically, it could've been implemented without generators at all. There's no point of trying to use asyncio for generator specific job. Just imagine that asyncio coroutines and generator coroutines - are different things.

Roughly saying, you should use asyncio only when you need to do some I/O operations parallely (like downloading multiple urls) and for nothing else. More about it.

answered Nov 20, 2017 at 5:09

Mikhail Gerasimov

40.1k20 gold badges126 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Martijn Pieters · Accepted Answer · 2017-11-23 16:28:40Z

3

await is the same thing as yield from, not yield; it delegates control of the generator to a next one.

You are not delegating control, so you would not use async def and await; you are merely sending data across to another generator. If an exception was being sent (with generator.throw()), this would not be passed on to your multiply() generator.

The point of async def is to create awaitable coroutines, and await only works with awaitables. The goal is that you can chain these together and still efficiently get information back to the event loop that drives them. yield is not used (or even legal) in an awaitable coroutine.

You may want to read this excellent blog post by Brett Cannon, a Python core developer, on how async def, await and yield from and @asyncio.coroutine tie together: How the heck does async/await work in Python 3.5?

You can combine a generator with a coroutine; as of Python 3.6 (which implemented PEP 525 - Asynchronous Generators:

async def multiply(factor):
    print(f"product table for {factor}")
    while True:
        sent_value = yield
        result = factor * sent_value
        print(f"{factor} x {sent_value} = {result}")

Note that I didn't replace the yield here. You'd use it like this:

async def product_table(coro):
    await coro.send(None) # start coroutine
    for value in range(1, 11):
        await coro.send(value)

There is little point in making this a coroutine, you are not using any awaitables in the product_table() generator. But say you wanted to send this data to a network socket instead:

async def multiply(factor):
    await socket.send(f"product table for {factor}")
    while True:
        sent_value = yield
        result = factor * sent_value
        await socket.send(f"{factor} x {sent_value} = {result}")

Now you have something that an event loop could manage for you, running multiple such tasks in parallel, sending multiplication tables to multiple clients as they are ready to receive the data.

edited Nov 23, 2017 at 16:28

answered Nov 18, 2017 at 17:36

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

3 Comments

Escualo Over a year ago

My thinking was upside-down. So, the coroutine can receive the generator (which may or may not be asynchronous) and delegate to it on every "iteration". Kind of?

Martijn Pieters Over a year ago

@Escualo: kind of. await works just like yield from, but has the added restriction that you have to have an awaitable object; this to prevent you from accidentally mixing old-skool generators and coroutines.

Martijn Pieters Over a year ago

@Escualo: (and this gives Python room to optimise for coroutines being stacked together too).

Keith · Accepted Answer · 2017-11-20 05:39:05Z

2

When you use async def you are creating a new type of first-class object, a coroutine object. These have to be driven by something, an asynchronous event loop. The await needs to call a coroutine, where it may be suspended. These are usually used for IO.

David Beazley wrote a nice framework for this, called curio. You can see a basic example echo server.

answered Nov 20, 2017 at 5:39

Keith

43.2k11 gold badges62 silver badges77 bronze badges

Collectives™ on Stack Overflow

Implement coroutine in terms of async def

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related