how to iterate through n amounts of lists in parallel?

Question

There are already many answers to 'how to iterate through 2 lists in parallel'. However, I am trying to figure out how to iterate through an arbitrary amount of lists.

For example, the following requires me to know beforehand how many lists i have

for f, b in zip(foo, bar):
    print(f, b)

But I am using os.listdir() to get an N amount of subfolders, and then I want to iterate through all those subfolder in parallel.

for (every nth element) in (each subfolder found):
    do something

Currently I have always done this manually but I'd really like to know how to do this more elegantly, i.e. without any knowledge of how many lists to iterate through.

Thanks!

Arne · Accepted Answer · 2018-02-13 08:40:10Z

4

You can still use zip, since it accepts more than two arguments. Given a list of lists, where you want to iterate through their i'th arguments in a loop:

lists = [[1, 2, 3, 4, 5], [1, 2, 3, 4, 5], [1, 2, 3, 4, 5]]
for parallel in zip(*lists):
    print(parallel)

prints:

(1, 1, 1)
(2, 2, 2)
(3, 3, 3)
(4, 4, 4)
(5, 5, 5)

Or, with a variant of your problem description:

import os

subdirs = [os.listdir(f) for f in os.listdir(my_dir) if os.path.isdir(os.path.join(my_dir, f))]
for parallel in zip(*subdirs):
    # work on them in parallel

It should be noted that zip cuts off on the end of the shortest list it gets. If you want to pad every list with Nones instead, you should use itertools.zip_longest, like so:

from itertools import zip_longest
# get subdirs same as before
for parallel in zip_longest(*subdirs):
    print([f for f in parallel if f is not None])  # don't show padded entries

edited Feb 13, 2018 at 8:40

answered Feb 9, 2018 at 13:16

Arne

20.7k11 gold badges101 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Daniel Lee · Accepted Answer · 2018-02-09 13:16:03Z

2

You could just unpack the list of lists:

list_of_lists = [[1,2,3], [2,3,4], [5,6,7]]

for x in zip(*list_of_lists):
    for y in x:
        print(y)
1
2
4
2
3
5
3
4
6

answered Feb 9, 2018 at 13:16

Daniel Lee

8,1994 gold badges31 silver badges47 bronze badges

Comments

FHTMitchell · Accepted Answer · 2018-02-09 13:17:08Z

1

You don't need to know how many lists you'll have in advance:

for subdirs in zip(*(os.listdir(p) for p in root_dirs)):

for paths in zip(*(os.listdir(p) for p in root_dirs))::
    do_something(paths)  # paths is a tuple like (paths[0], paths[1], ...)

answered Feb 9, 2018 at 13:17

FHTMitchell

12.2k2 gold badges40 silver badges50 bronze badges

Comments

FlyingTeller · Accepted Answer · 2018-02-09 13:14:38Z

0

for your list of subfolders:

subfolders = os.listdir(some_path)

you can create a second for loop around your first one:

# iterate through subfolders of some_path
for subfolder in subfolders:
    #iterate through elements in subfolder
    for element in os.listdir(os.path.join(some_path, subfolder):
        # you take it from here

You might also want to take a look at os.walk to recursively iterate through a directory tree

answered Feb 9, 2018 at 13:14

FlyingTeller

21k3 gold badges49 silver badges62 bronze badges

Collectives™ on Stack Overflow

how to iterate through n amounts of lists in parallel?

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related