Calling multiprocessing pool on an instance method in Python

Question

I've a Driver class which has an instance method write. I'm trying to use Python multiprocessing to apply this instance method on objects in queue. Here's my code.

class Driver:

def __init__(self, arg1, arg2, arg3):
   #initialize

def write(self, key):
    # do something

Here's how I'm calling it on a collection.

def write_to_db(key, d=Driver(arg1=None, arg2=None, arg3=None)):
    d.write(key)

def main():
  .........
   with multiprocessing.Pool(processes=8) as pool:
                    driver = Driver(arg1=arg1, arg2=arg2, arg3=arg3)
                    _ = pool.map(write_to_db, (arr, driver))

I get the following error:

TypeError: can't pickle _thread.RLock objects

How can I get this right?

can you please show more of your code? I'm missing for example the important if __name__ == "__main__" clause and if I see it right, your main()-function is outside of the class Driver ,right? — TornaxO7
– TornaxO7, Commented Aug 7, 2019 at 20:40
In your pool.map, write_to_db will only be run twice, once with arr as the argument, and once with driver. Is that really what you intended? — Roland Smith
– Roland Smith, Commented Aug 23, 2019 at 21:23

user11751463 · Accepted Answer · 2019-08-23 21:05:26Z

2

The simple answer is you can't, if you have shared state the object won't be picklable and python will throw an error. The long answer, to do this, you have to instantiate the object inside a process in the pool.

You can instantiate the pool in this way,

  with mp.Pool(processes=processes, initializer=_worker_init, initargs=(config,)) as pool:
        results = pool.map(_worker_apply, resources)

config passes if you want to pass configurations resources is the collection here on which the individual processes will work.

    def _worker_init(config):
        # initialize the object here


    def _worker_apply(payload):
        process = mp.current_process()
        return process.simulator.apply(payload)# implement the apply function to manage state of the object

Collectives™ on Stack Overflow

Calling multiprocessing pool on an instance method in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related