Parallel Python - create objects into another class in parallel execution

Question

I'm trying to do something pretty simple with Parallel Python. I would like to be able to create an object from a class I've created inside another method from a class use to do a job in parallel. Here is a basic example of what I would like to make it work :

import pp

class TestClass(object):
    def __init__(self): 
        pass
    def doSomething (self, number) : 
        print number**2

class PPTask (object) :
    def __init__ (self) : 
        pass
    def ppTask(self, number = 1) :
        sum = 0
        sum += number
        tc = TestClass()
        tc.doSomething(sum)
        return sum

if __name__ == '__main__' : 
    job_server = pp.Server()
    job_list = []
    results = []
    for i in xrange(10) : 
        pt = PPTask()
        job_list.append(job_server.submit(pt.ppTask, (1,), globals = globals()))
    for job in job_list : 
        results.append(job())
    for result in results :
        print result

This raise NameError: global name 'TestClass' is not defined and I didn't find any solution to pass it or reuse it in the ppTask method.

Any help would be greatly appriciated.

Thank you in advance

Brendan Wood · Accepted Answer · 2012-04-03 16:52:19Z

1

One solution would be to tell the job server to import the source module itself for each job you submit. For example, if your script above were called pptest.py, you could create the jobs like so:

job_list.append(job_server.submit(pt.ppTask, (1,), modules=('pptest',)))

And within ppTask, you could instantiate TestClass like so:

tc = pptest.TestClass()

So overall, the code would look like this:

import pp

class TestClass(object):
    def __init__(self):
        pass
    def doSomething (self, number) :
        print number**2

class PPTask (object) :
    def __init__ (self) :
        pass
    def ppTask(self, number = 1) :
        sum = 0
        sum += number
        tc = pptest.TestClass()
        tc.doSomething(sum)
        return sum

if __name__ == '__main__' :
    job_server = pp.Server()
    job_list = []
    results = []
    for i in xrange(10) :
        pt = PPTask()
        job_list.append(job_server.submit(pt.ppTask, (1,), modules=('pptest',)))
    for job in job_list :
        results.append(job())
    for result in results :
        print result

answered Apr 3, 2012 at 16:52

Brendan Wood

6,4903 gold badges32 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Sebastien Givord Over a year ago

Thank you, it works well. I still don't understand how globals is handle by th submit method of PP. I did try to pass the TestClass in a dictionnary via globals = {'TestClass': <class 'main.TestClass'>} and I can't figure out why this doesn't works too.

Brendan Wood Over a year ago

Yeah, I tried that too, but couldn't get that to work either. It seems like it should, but I don't know enough about pp to say that confidently. I had never run into this before because I usually put separate classes in separate module files, and just use the modules keyword argument to Server.submit() to tell the job what to import.

Collectives™ on Stack Overflow

Parallel Python - create objects into another class in parallel execution

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related