python multiprocessing write data to the same list

Question

I want to write data to the same list via python multiprocessing, I do interprocess data sharing via mp.manager.list. The code is shown below, this is just a demo, I want to add the same numbers to the same list. However, counter can be increased, but grp remains the same. Where is the problem?

import multiprocessing as mp
import random
import time

import numpy as np

class A:
    def __init__(self):
        self.raw = [random.randint(1, 4) for _ in range(100)]
        self.manager = mp.Manager()
        self.grp = self.manager.list([[1], [2], [3], [4]])
        self.use_cpu_num = 2
        self.counter = self.manager.Value('i', 0)

    def run(self):
        subsets = np.array_split(self.raw, self.use_cpu_num)
        subsets = [each.tolist() for each in subsets]
        process = []
        for i in range(self.use_cpu_num):
            process.append(mp.Process(target=self.process, args=(subsets[i], )))
        for each in process:
            each.start()
        for each in process:
            each.join()
            each.close()
        print(self.grp)

    def process(self, subset):
        for each in subset:
            for i in range(len(self.grp)):
                each_grp = self.grp[i]
                if each in each_grp:
                    self.counter.set(self.counter.value + 1)
                    self.grp[i].append(each)
                    print(self.counter.value)



if __name__ == '__main__':
    a = A()
    a.run()

I tried using mp.Lock(), but that doesn't share data between different processes.

Ahmed AEK · Accepted Answer · 2022-11-22 13:52:00Z

Put it this way, self.grp is a managed object, any change on it using self.grp.append or self.grp[i] = x will be transferred to the manager process.

The objects inside self.grp are not managed, any change to them will not be transferred to the manager, you only get a copy of them when you use self.grp[i].

In order to allow modifications to the lists inside self.grp to propagate, those lists must themselves be manager.list object, and nesting managed objects is not supported for versions of python below 3.6

self.grp = self.manager.list([self.manager.list(x) for x in ([1], [2], [3], [4])])

If you are only storing numbers you can pass multiprocessing.Array which can be wrapped as a numpy ndarray for convenience, but you cannot append to it, and must know the size beforehand.

Edit: on windows you will get an error when trying to pickle the self.manager object, so i modified it out of the class in the example below.

import multiprocessing as mp
import random
import time

import numpy as np
class A:
    def __init__(self):
        self.raw = [random.randint(1, 4) for _ in range(100)]
        self.grp = manager.list([manager.list(x) for x in ([1], [2], [3], [4])])
        self.use_cpu_num = 2
        self.counter = manager.Value('i', 0)

    def run(self):
        subsets = np.array_split(self.raw, self.use_cpu_num)
        subsets = [each.tolist() for each in subsets]
        process = []
        for i in range(self.use_cpu_num):
            process.append(mp.Process(target=self.process, args=(subsets[i], )))
        for each in process:
            each.start()
        for each in process:
            each.join()
            each.close()
        print([list(x) for x in self.grp])

    def process(self, subset):
        for each in subset:
            for i in range(len(self.grp)):
                each_grp = self.grp[i]
                if each in each_grp:
                    self.counter.set(self.counter.value + 1)
                    self.grp[i].append(each)
                    print(self.counter.value)



if __name__ == '__main__':
    manager = mp.Manager()
    a = A()
    a.run()

Collectives™ on Stack Overflow

python multiprocessing write data to the same list

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related