How to re-use pool workers in multiprocessing code?

Question

In the below code, I get an error about "can't get attribute 'f' on module main". I know how to fix it: bring the pool line and the result line both to just above result 2.

My question is why the code in its current form doesn't work. I am working with more complicated code where I have to use parallel processing inside of two different separate for loops. Right now, I have in each iteration of each for loop, pool=mp.Pool(3). I read online that this is bad because in each iteration, I am creating more Pool "workers." How can I put pool = mp.Pool(3) on the outside of the iteration and then use the same Pool workers in all of the different areas of my code that I need to?

For the record, I am using a mac to run my code.

import numpy as np
import multiprocessing as mp

x = np.array([1,2,3,4,5,6])

pool = mp.Pool(3)

def f(x):
    return x**2

result = pool.map(f,x)

def g(x):
    return x + 1

result2 = pool.map(g,x)
print('result=',result,'and result2=',result2)

Creating the Pool creates the necessary subprocesses by forking (on Mac OS) at this point. This means the forked children haven't yet executed the creation of "f" but instead wait for tasks from main process. — Michael Butscher
– Michael Butscher, Commented Oct 1, 2019 at 23:02
@MichaelButscher I am truly confused. So are you saying there is no way to do what I want to do above, where I define pool once and then can use pool anywhere subsequent in my code? Right now, I am defining pool in a for loop, so in each iteration, pool = mp.Pool(3) is run... — layman
– layman, Commented Oct 2, 2019 at 1:23
never ever define a pool in a loop because it keeps spanwing pools and pool-workers in uncontrolled manner and thus exhausting your RAM memory leaving nothing for other programs and eventually result in a computer crash. After you created a pool (def main from Michael Butsch's answer) you can use a while-loop for daemon-like activities and re-using pool members. — ZF007
– ZF007, Commented Feb 10, 2020 at 10:50

Michael Butscher · Accepted Answer · 2019-10-02 03:07:48Z

1

When using "fork" method for creating subprocesses (default for Mac OS) the processes are forked (basically copied) when the Pool is created. This means in your code the forked children haven't yet executed the creation of f but instead wait for tasks from main process.

First of all you should not execute "active" code (other than defining functions, classes, constants) directly in the script but move it to functions. Your code can then look like:

import numpy as np
import multiprocessing as mp


def f(x):
    return x**2

def g(x):
    return x + 1

def main():
    x = np.array([1,2,3,4,5,6])

    pool = mp.Pool(3)

    result = pool.map(f,x)
    result2 = pool.map(g,x)
    print('result=',result,'and result2=',result2)

# Should be nearly the only "active" statement
main()

Or maybe better in your case, I guess:

import numpy as np
import multiprocessing as mp


def f(x):
    return x**2

def g(x):
    return x + 1

def proc_f():
    global x, pool
    return pool.map(f,x)

def proc_g():
    global x, pool
    return pool.map(g,x)

def main():
    global x, pool
    x = np.array([1,2,3,4,5,6])

    pool = mp.Pool(3)

    result = proc_f()
    result2 = proc_g()
    print('result=',result,'and result2=',result2)

# Should be nearly the only "active" statement
main()

edited Oct 2, 2019 at 3:07

answered Oct 2, 2019 at 2:58

Michael Butscher

11k4 gold badges28 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

layman Over a year ago

Thank you. It's been a few days since you posted your answer, but I have been able to rework my code. From your examples, I understand now that any functions I want the pool workers to use must be defined before the line mp.Pool(). I have restructured my code so that I am only calling the Pool() function once.

Collectives™ on Stack Overflow

How to re-use pool workers in multiprocessing code?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related