Parallel Python global name error

Question

Hey im trying to run a simple code that adds a list of numbers together at the same time using Parallel Python

import sys, time
import pp
import numpy
x = numpy.arange(-20.0,20.0,0.5)
def function(raw_input):
    f = 0
    for i in numpy.arange(len(x)):
        f+=1
    a=raw_input[0]
    b=raw_input[1]
    c=raw_input[2]
    d=raw_input[3]
    print len(x)
    return (a+b+c+d)+f
# tuple of all parallel python servers to connect with
ppservers = ()
#ppservers = ("10.0.0.1",)

if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"

start_time = time.time()

# The following submits 4 jobs and then retrieves the results
puts = ([1,2,3,4], [3,2,3,4],[4,2,3,6],[2,3,4,5])

jobs = [(raw_input, job_server.submit(function,(raw_input,), (), ("numpy",))) for raw_input in puts]
for raw_input, job in jobs:
    print "Sum of numbers", raw_input, "is", job()

print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()

so basically i want it to add [1,2,3,4] together at the same time its adding [3,2,3,4],[4,2,3,6],[2,3,4,5]. And add "f" which is the length of x (which is 80) to all the answers. the out put should look like this:

Starting pp with 4 workers

Sum of numbers [1, 2, 3, 4] is 90

Sum of numbers [3, 2, 3, 4] is 92

Sum of numbers [4, 2, 3, 6] is 95

Sum of numbers [2, 3, 4, 5] is 94

Time elapsed: 0.394000053406 s

Job execution statistics:

job count | % of all jobs | job time sum | time per job | job server

     4 |        100.00 |       1.4380 |     0.359500 | local

Time elapsed since server creation 0.442999839783

the problem im having is that with x outside of "function" the shell comes back with global name 'x' not defined, but if you put x into the shell it returns the full array of x.

im confused why its clearly defined enough to give me back "x" when i put it in the shell but the job doesn't find x or anything else outside of the function definition.

Peter de Rivaz · Accepted Answer · 2012-02-14 19:55:19Z

1

I think the problem is that x is defined on the job server (which is the one running in the shell), but not in the job workers because the workers only have access to the variables in their inputs.

You should be able to fix it by passing in x as an additional argument when you submit the jobs.

As far as I can see, the calculation of f will always have the result of len(x) so you could simply pass in the value for f instead:

import sys, time
import pp
import numpy
x = numpy.arange(-20.0,20.0,0.5)
def function(raw_input,f):
    a=raw_input[0]
    b=raw_input[1]
    c=raw_input[2]
    d=raw_input[3]
    return (a+b+c+d)+f
# tuple of all parallel python servers to connect with
ppservers = ()
#ppservers = ("10.0.0.1",)

if len(sys.argv) > 1:
    ncpus = int(sys.argv[1])
    # Creates jobserver with ncpus workers
    job_server = pp.Server(ncpus, ppservers=ppservers)
else:
    # Creates jobserver with automatically detected number of workers
    job_server = pp.Server(ppservers=ppservers)
print "Starting pp with", job_server.get_ncpus(), "workers"

start_time = time.time()

# The following submits 4 jobs and then retrieves the results
puts = ([1,2,3,4], [3,2,3,4],[4,2,3,6],[2,3,4,5])

jobs = [(raw_input, job_server.submit(function,(raw_input,len(x)), (), ("numpy",))) for raw_input in puts]
for raw_input, job in jobs:
    print "Sum of numbers", raw_input, "is", job()

print "Time elapsed: ", time.time() - start_time, "s"
job_server.print_stats()

answered Feb 14, 2012 at 19:55

Peter de Rivaz

33.6k4 gold badges51 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user991926 Over a year ago

Thanks for the reply,i am aware that f will always be the same, i was trying to make a simple example of pulling a variable from outside the function into the calculations. the real objective is actually a different code with something like this build into it, and it has variables that change over each iteration and the calculation inside the parallel job is supposed to append into an array that is also outside the function so that's why i didn't just put the x inside.this might work however if i can make an array or a list part of the arguments i think. Thanks for the help

Collectives™ on Stack Overflow

Parallel Python global name error

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related