Python multiprocessing: Pool of custom Processes

Question

I am subclassing the Process class, into a class I call EdgeRenderer. I want to use multiprocessing.Pool, except instead of regular Processes, I want them to be instances of my EdgeRenderer. Possible? How?

Are you trying to write your code to use multi threading this way? — Brian Gianforcaro
– Brian Gianforcaro, Commented Apr 11, 2009 at 23:27

mskoh52 · Accepted Answer · 2016-12-01 18:50:06Z

5

This seems to work:

import multiprocessing as mp

ctx = mp.get_context()  # get the default context

class MyProcess(ctx.Process):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        print("Hi, I'm custom a process")

ctx.Process = MyProcess  # override the context's Process

def worker(x):
    print(x**2)

p = ctx.Pool(4)
nums = range(10)
p.map(worker, nums)

answered Dec 1, 2016 at 18:50

mskoh52

1503 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ram Rachum · Accepted Answer · 2009-04-12 14:24:46Z

3

From Jesse Noller:

It is not currently supported in the API, but would not be a bad addition. I'll look at adding it to python2.7/2.6.3 3.1 this week

answered Apr 12, 2009 at 14:24

Ram Rachum

89.6k88 gold badges258 silver badges390 bronze badges

Comments

David Berger · Accepted Answer · 2009-04-12 01:26:06Z

2

I don't see any hook for it in the API. You might be able to get away with replicating your desired functionality by using initializer and initargs argument. Alternately, you can build the functionality into the callable object that you use for mapping:

class EdgeRenderTask(object):
    def op1(self,*args):
        ...
    def op2(self,*args):
        ...
p = Pool(processes = 10)
e = EdgeRenderTask()
p.apply_async(e.op1,arg_list)
p.map(e.op2,arg_list)

answered Apr 12, 2009 at 1:26

David Berger

12.9k6 gold badges41 silver badges53 bronze badges

1 Comment

catwalker333 Over a year ago

You can't pass a class as an argument to p.map, you'll get pickle error. The only way I've found around this is to implement a sort of custom pool.map....setting up and feeding the processes myself. Unfortunately it's gonna take more lines.

Collectives™ on Stack Overflow

Python multiprocessing: Pool of custom Processes

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related