5

I've been reading and re-reading the IPython documentation/tutorial, and I can't figure out the issue with this particular piece of code. It seems to be that the function dimensionless_run is not visible to the namespace delivered to each of the engines, but I'm confused because the function is defined in __main__, and clearly visible as part of the global namespace.

wrapper.py:

import math, os

def dimensionless_run(inputs):
    output_file = open(inputs['fn'],'w')
    ...
    return output_stats

def parallel_run(inputs):
    import math, os  ## Removing this line causes a NameError: global name 'math'
                     ## is not defined.
    folder = inputs['folder']
    zfill_amt = int(math.floor(math.log10(inputs['num_iters'])))
    for i in range(inputs['num_iters']):
        run_num_str = str(i).zfill(zfill_amt)
        if not os.path.exists(folder + '/'):
            os.mkdir(folder)
        dimensionless_run(inputs)
    return

if __name__ == "__main__":
    inputs = [input1,input2,...]
    client = Client()
    lbview = client.load_balanced_view()
    lbview.block = True
    for x in sorted(globals().items()):
        print x
    lbview.map(parallel_run,inputs)

Executing this code after ipcluster start --n=6 yields the sorted global dictionary, including the math and os modules, and the parallel_run and dimensionless_run functions. This is followed by an IPython.parallel.error.CompositeError: one or more exceptions from call to method: parallel_run, which is composed of a large number of [n:apply]: NameError: global name 'dimensionless_run' is not defined, where n runs from 0-5.

There are two things I don't understand, and they're clearly linked.

  1. Why doesn't the code identify dimensionless_run in the global namespace?
  2. Why is import math, os necessary inside the definition of parallel_run?

Edited: This turned out not be much of a namespace error at all--I was executing ipcluster start --n=6 in a directory that didn't contain the code. To fix it, all I needed to do was execute the start command in my code's directory. I also fixed it by adding the lines:

    inputs = input_pairs
    os.system("ipcluster start -n 6") #NEW
    client = Client()
    ...
    lbview.map(parallel_run,inputs)
    os.system("ipcluster stop")       #NEW

which start the required cluster in the right place.

Improve this question

1 Answer 1

Reset to default
8

This is mostly a duplicate of Python name space issues with IPython.parallel, which has a more detailed answer, but the gist:

When the Client sends parallel_run to the engine, it just sends that function, not the entire namespace in which the function is defined (the __main__ module). So when running the remote parallel_run, lookups to math or os or dimensionless_run will look first in locals() (what has been defined already in the function, i.e. your in-function imports), then in the globals(), which is the __main__ module on the engine.

There are various approaches to making sure names available on the engines, but perhaps the simplest is to explicitly define/send them to the engines (the interactive namespace is __main__ on the engines, just like it is locally in IPython):

client[:].execute("import os, math")
client[:]['dimensionless_run'] = dimensionless_run

prior to making your run, in which case everything should work as you expect.

This is an issue unique to modules defined interactively / in a script - It does not come up if this file is a module instead of a script, e.g.

from mymod import parallel_run
lbview.map(parallel_run, inputs)

In which case the globals() is the module globals, which are generally the same everywhere.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Both of these solutions work partway, in that they add dimensionless_run to the namespace. However, the functions and classes that are called by dimensionless_run are now the ones that come up as missing from the namespace. There has to be an efficient/pythonic way of sending this data to the engines, without writing out a line of code for every function that needs to be passed along.
I do one of three approaches, all of which result in complete namespaces: 1. define the functions remotely in the first place with the %%px magic (I do this when using the IPython notebook). 2. use modules, where this issue never comes up. 3. make all functions defined locally, but intended to be used remotely, never resolve names outside from globals (100% of names are function arguments or imports within the function). Yet another option, when working from a script like yours, is to simply push globals() (excluding the Client object itself, of course).
Hmm, I moved all the function-definition code into a module, and moved the if __name__ == "__main__" from perceptions_wrapper import parallel_run import time, sys from IPython.parallel import Client
(Sorry for the double post) Hmm, I moved all the function-definition code into a module, and moved the if __name__ == "__main__"... code to its own script that depends only on: from wrapper import parallel_run from IPython.parallel import Client and I still get the same error. If I remove the import math, os line from parallel_run (which it seems like I should be able to do if I'm importing from the module), they become the first objects missing in the namespace.
That's bizarre, and not been my experience. If it didn't work, then calling apply with any module function (e.g. numpy.linalg.norm or json.dumps) wouldn't work either, and they clearly do, so I don't know what is different about what you are doing without seeing it. Please provide a complete example that reproduces your issue, and perhaps bring it over to ipython-dev or GitHub, which are more appropriate venues to discuss such things in detail.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.