Map, filter and itertools for composing asynchronous iterators

Question

Does Python support functional-style operations on asynchronous iterators? I know that I can use map, filter and itertools to lazily transform and consume data coming from normal generators:

from itertools import accumulate, takewhile

def generator():
    a, b = 1, 1
    while True:
        yield a
        a, b = b, a + b

# create another iterator, no computation is started yet:
another_iterator = takewhile(lambda x: x < 100, accumulate(generator()))
# start consuming data:
print(list(another_iterator))
# [1, 2, 4, 7, 12, 20, 33, 54, 88]

Now, the same thing is not supported on Python 3.6's asynchronous generators/iterators because of course they do not implement the normal iterator protocol:

async def agenerator():
    a, b = 1, 1
    while True:
        yield a
        a, b = b, a + b

accumulate(agenerator())

TypeError: 'async_generator' object is not iterable

Is there some kind of async map or async itertools to achieve the similarly lazy behaviour in Python 3.6/3.7?

Mikhail Gerasimov · Accepted Answer · 2018-05-31 13:21:56Z

8

Most full asynchronous version of itertools I saw is aiostream module. Your example will be:

import asyncio
from aiostream.stream import takewhile, accumulate, list as alist


async def agenerator():
    a, b = 1, 1
    while True:
        yield a
        a, b = b, a + b


async def main():
    another_iterator = takewhile(
        accumulate(agenerator()),
        lambda x: x < 100, 
    )

    res = await alist(another_iterator)

    print(res)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

answered May 31, 2018 at 13:21

Mikhail Gerasimov

40.1k20 gold badges126 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Norrius Over a year ago

Thanks, this looks very interesting, especially the async version of list. I'm looking into the library, it seems fairly mature although it's at 0.3.0.

Rémi Benoit Over a year ago

Is it still the best way to achieve this ? Is there no talk of including a standard way of handling async generators ?

Collectives™ on Stack Overflow

Map, filter and itertools for composing asynchronous iterators

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related