How to set an instance attribute in parallel in a Python class?

Question

I want to set an instance attribute by running an instance method in parallel. Let's say the attribute is initially an empty dictionary called d, and I want to update it in parallel by an instance method called update_d. I am currentely using multiprocessing.Pool:

from multiprocessing import Pool
import random


class A():

    def __init__(self, n_jobs):

        self.d = dict()
        self.n_jobs = n_jobs
        pool = Pool(self.n_jobs)
        pool.map(self.update_d, range(100))
        pool.close()

    def update_d(self, key):
        self.d[key] = random.randint(0, 100)


if __name__ == '__main__':
    a = A(n_jobs=4)
    print(a.d)

However, the attribute is not updated after running update_d in parallel. I understand that it's because multiprocessing.Pool always folks the instance to individual processes. But I want to know what is the recommended way to do this in Python? Note that I don't want to return anything from update_d, and we can assume that the code is written in a way that the individual processes won't conflict with each other.

Edit: I just use dictionary as an example. I need a solution that allows the attribute to be any type of variable, e.g. a Pandas dataframe.

quamrana · Accepted Answer · 2022-10-12 15:20:41Z

1

You may need a Manager to create a dict for you. I still don't know how well the updates will work, whether there will be any race conditions.

from multiprocessing import Pool, Manager
import random


class A():

    def __init__(self, n_jobs, manager):
        self.d = manager.dict()
        self.n_jobs = n_jobs
        pool = Pool(self.n_jobs)
        pool.map(self.update_d, range(100))
        pool.close()

    def update_d(self, key):
        self.d[key] = random.randint(0, 100)


if __name__ == '__main__':
    with Manager() as manager:
        a = A(n_jobs=4, manager=manager)
        print(a.d)

answered Oct 12, 2022 at 15:20

quamrana

39.5k13 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Shaun Han Over a year ago

Does Manager only works for dictionaries? In my actualy case, I have a Pandas Dataframe object to be updated. I tried self.df = Manager().Dataframe() but it doesn't work.

quamrana Over a year ago

No, I think that Manager only has python native structures. (That is just list and dict)

Collectives™ on Stack Overflow

How to set an instance attribute in parallel in a Python class?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related