Python: async over a list

Question

I am trying to write my first piece of asynchronous code. I've simplified my use-case down and have two scripts - 'main' and 's3_accessor'. The main script builds out a list of values that I then want to pass to the 'build_inventory' function.

The list has hundreds of values, so synchronous processing is slow.

main.py

import s3_accessor # another .py script
import asynchio

async def main():
     some_list [a, b, c, d, e, f, g]
     
     for val in some_list:
          inventory_df = s3_accessor.build_inventory(val)
          s3_accessor.write_df_to_s3('some_key', inventory_df)


if __name__ == "__main__":
     loop = asyncio.get_event_loop()
     loop.run_until_complete(main())

s3_accessor.py

async def build_inventory(val):
     ...

Question:

How would I modify this code to make it such that the iteration over the list is asynchronous?

I tried wrapping the entire for loop section in await asyncio.wait() but that doesn't appear to be the correct approach.

await asyncio.wait() is to avoid race condition when two processes use shared data. It is not directly related to being asynchronous or not. — Farshid
– Farshid, Commented Jul 18, 2022 at 2:18
What exactly are you trying to make asynchronous. Do you want all the values in the list to happen at the same time? Is it okay if multiple copies of build_inventory are running simultaneously? Is it okay if multiple copies of write_df_to_s3 are running simultaneously? — Frank Yellin
– Frank Yellin, Commented Jul 18, 2022 at 2:19

Frank Yellin · Accepted Answer · 2022-07-18 02:26:34Z

2

If, as I asked above, you're trying to handle all the values at the same time, and are okay with the calls to s3_accessor happening concurrently:

async def handle_one_value(val):
    inventory_df = s3_accessor.build_inventory(val)
    s3_accessor.write_df_to_s3('some_key', inventory_df)

async def main():
     some_list = [a, b, c, d, e, f, g]
     wait asyncio.gather(*(handle_one_value(val) for val in some_list))

answered Jul 18, 2022 at 2:26

Frank Yellin

11.6k1 gold badge15 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python: async over a list

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related