2

There are two functions:

def tibidam(..., foo_dyn, index):
    print("(" + str(index) + ") B:", foo_dyn)

    for i in range(...):
        for j ...
            if j not in foo_dyn:
                foo_dyn[ep] = j

    print("(" + str(index) + ") A:", foo_dyn)

def tata(..., foo):
    foo_dyn = Array('i', len(foo))
    
    foo_dyn = foo

    with Pool(processes=4) as pool:
        pool.starmap(tibidam, [(..., foo_dyn, i) 
            for i in range(4)])
    
    return foo

Output (formatted):

foo  : [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(1) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(2) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(3) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) A: [27, 1,  2,  3,  64, 5,  6,  7,  80, 9,  10, 11]
(2) A: [0,  1,  64, 3,  4,  5,  13, 7,  8,  9,  92, 11]
(3) A: [0,  1,  2,  31, 4,  5,  6,  73, 8,  9,  10, 18]
(1) A: [0,  18, 2,  3,  4,  27, 6,  7,  8,  99, 10, 11]
...

Expected output (formatted):

foo  : [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(1) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(2) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(3) B: [0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11]
(0) A: [27, 1,  2,  3,  64, 5,  6,  7,  80, 9,  10, 11]
(2) A: [27, 1,  55, 3,  64, 5,  13, 7,  80, 9,  92, 11]
(3) A: [27, 1,  55, 31, 64, 5,  13, 73, 80, 9,  92, 18]
(1) A: [27, 87, 55, 31, 64, 88, 13, 73, 80, 99, 92, 18]
...

How I can change foo_dyn in all workers, whenever foo_dyn changes? It seems, that pool.starmap(...) creates a copy of the foo_dyn, for each process... No, I want pass foo_dyn to the pool only once. But, again, without using global variables, at all.

AFAIK, multiprocessing.Pool supports initializer and initargs arguments: I can write own initializer:

_init(foo):
    global foo_dyn

    foo_dyn = foo

, but it uses global variable foo_dyn (by the way, using _init function doesn't solve the problem). In passing, I saw a few questions, with almost the same trouble. However, all solutions were associated with using global variables.

Improve this question
12
  • 1
    Shared memory : multiprocessing module provides Array and Value objects to share data between processes. Array: a ctypes array allocated from shared memory. Value: a ctypes object allocated from shared memory. Commented Mar 15, 2020 at 6:12
  • docs.python.org/3.8/library/… Commented Mar 15, 2020 at 6:13
  • docs.python.org/3.8/library/… Commented Mar 15, 2020 at 6:14
  • Oh, @Joe, thanks you! :) I've seen that, but I'm more interested in a solution compatible with older Python versions. For example, it can be implemented with using multiprocessing.Process class. But, due to inefficiency, for my case, of this approach, I want use multiprocessing.Pool class (without using global variables). Commented Mar 15, 2020 at 11:36
  • 1
    @MaximTemny: Sorry, I phrased that very poorly. I mean to say that it should work to pass such objects, and you should phrase your question as “Why isn’t this working?” with the usual complete reproducer. (For example, your code shows foo_dyn being reassigned to be just foo, which is surely wrong.) Commented Mar 17, 2020 at 21:15

1 Answer 1

Reset to default
1

I found the solution, without using the global variables:

from multiprocessing import Pool, Manager

def tibidam(..., foo_dyn, index):
    for i in range(...):
        for j ...
            if j not in foo_dyn:
                foo_dyn[ep] = j

def tata(..., foo):
    foo_dyn = Manager().list(foo)

    with Pool(processes=4) as pool:
        pool.starmap(tibidam, [(..., foo_dyn, i)
            for i in range(4)])

    return foo_dyn

Thank you all! :>

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.