4

The multiprocessing docs say it is better to inherit (from the ancestor process) than to pickle/unpickle a shared resource (and hence to avoid using pipes/queues).

However, it also says it is better to explicitly pass resources, rather than to share as global resources.

So, how do you pass a shared resource, e.g. to a map function?


This example shows that a shared global works (but is not explicit),

import multiprocessing.sharedctypes
import numpy as np

def task(i):
    global x
    x[i] = i

sharedresource = multiprocessing.sharedctypes.RawArray('b', 10)
x = np.frombuffer(sharedresource, dtype=np.uint8)

with multiprocessing.Pool(4) as pool:
    pool.map(task, range(10)) # implicitly modify x

print(x) # [0 1 2 3 4 5 6 7 8 9]

but explicit passing does not work (presumably pickling in some queue has broken the shared connection):

def task2(i, array):
    array[i] = 7

with multiprocessing.Pool(4) as pool:
    pool.starmap(task2, ((i,x) for i in range(10))) # explicit fail

print(x) # still [0 1 2 3 4 5 6 7 8 9]
1
  • stackoverflow.com/a/1721911 suggests passing the resource as an initargs argument to Pool and writing an initialize function to assign it to a new global under each process. That might be more portable, but isn't terribly clean looking. Commented Mar 1, 2018 at 5:54

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.