Synchronization issue in IPython parallel computing

Question

I recently want to develop a HTTP server frontier to wrap my ipcontroller/ipengine clustering program. The server is a simple derived from BaseHTTPServer. When the server receives a HTTP Get request, its do_GET method will call several mec.execute() method to finish the job. Here is the code sample.

do_GET
{
b = parameter
mec.scatter("a", b)
mec.execute("c=fun(a)")
d = mec.gather("c")
write d
}

Will I face Synchronization problem at the statement mec.execute("c=fun(a)")? From my guessing, a variable "c" will be created at each ipengie with value "fun(a)". If two threads call do_Get method simultaneously with different parameter, what will be the value of "c" at each of ipengine.

minrk · Accepted Answer · 2011-12-17 16:26:48Z

If you can express the task as single parallel function call, then you should be safe, because no other requests can sneak in between (and engine globals need not be touched), e.g:

from IPython import parallel

rc = parallel.Client()
view = rc[:]

@view.parallel(block=True)
def pfun(a):
    """each engine will get a chunk of a, not the whole thing"""
    c = fun(a)
    return c

# a will be scattered and c will be gathered
c = pfun(a)

But if not, then the easiest solution is probably to ensure that you don't have name collisions across jobs by giving your variables for a given request a unique suffix with a UUID:

import uuid
suffix = str(uuid.uuid4()).replace('-','') # remove '-' so we have a valid identifier
a_name = "a_" + suffix
c_name = "c_" + suffix
mec.scatter(a_name, b)
mec.execute("%s = fun(%s)" % (c_name, a_name))
d = mec.gather(c_name)

Collectives™ on Stack Overflow

Synchronization issue in IPython parallel computing

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related