multiprocessing pool map returning the same values for each call

Question

I'm having a problem getting different return values from a pool.map call. Im running this simplified example on a macbook, in a jupyter notebook with python 3.9:

import multiprocessing as mp
import numpy as np

def gen_random_list(num):
    print(f"Num: {num}\n")
    ret_list = np.random.randint(0, 5, size=3).tolist()
    return ret_list
    
with mp.Pool() as pool:
    results = []
    results = pool.map(gen_random_list, range(4))
    print(results)

I keep getting the same list back for each call to gen_random_list. The complete output is this:

Num: 0
Num: 1
Num: 2
Num: 3




[[0, 3, 2], [0, 3, 2], [0, 3, 2], [0, 3, 2]]

Does anyone know why every call to gen_random_list is returning the same values (in this run: [0, 3, 2])? And how do I fix this so I get a different random list back each time?

Thanks Brett

The code as written does not run unless I put the multiprocessing part inside of if __name__ == '__main__'. Once I do that, I cannot reproduce your results, I just got [[0, 1, 2], [2, 0, 2], [2, 4, 3], [2, 1, 0]]. — Kraigolas
– Kraigolas, Commented Aug 9, 2022 at 15:38
It works as written in a jupyter notebook. The check for main, won't work in a notebook, since the notebook isn't the main process. — Brett Elliot
– Brett Elliot, Commented Aug 9, 2022 at 15:54
This is probably because the processes use the same seed to generate the numbers. Relevant stackoverflow.com/questions/73214631/… — Charchit Agarwal
– Charchit Agarwal, Commented Aug 9, 2022 at 17:13

Brett Elliot · Accepted Answer · 2022-08-09 17:39:25Z

0

Thanks @Charchit! That was it! Subtle for sure.... Here's a working example now:

import multiprocessing as mp
import numpy as np

def gen_random_list2(num):
    print(f"Num: {num}\n")
    # https://stackoverflow.com/questions/22994423/difference-between-np-random-seed-and-np-random-randomstate
    local_state = np.random.RandomState(num)
    ret_list = local_state.randint(0, 5, size=3).tolist()
    return ret_list
    
with mp.Pool(processes=processes) as pool:
    results = list(map(gen_random_list2, range(100)))
    print(results)

answered Aug 9, 2022 at 17:39

Brett Elliot

9621 gold badge6 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

multiprocessing pool map returning the same values for each call

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related