4

I had some problems with calling pip within the IPython REPL, and after a while I noticed that IPython doesn't use the same $PATH environment as my shell.

$ echo $PATH
/Users/jimmy/dev/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin

$ ipython
In [1]: !echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin:/Users/jimmy/dev/anaconda/bin

It does seem to scramble the path order, and I'm not quite sure what's wrong.

I'm using ZSH installed through oh-my-zsh as shell, if it is of help.

This is what sys.path() returns:

['',
 '/Users/jimmy/dev/anaconda/bin',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/sparsesvd-0.2.2-py2.7-macosx-10.5-x86_64.egg',
 '/Users/jimmy/dev/projects/pyresult',
 '/Users/jimmy/dev/work/gavagai/userdata',
 '/Users/jimmy/dev/work/gavagai/gavapi',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python27.zip',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-darwin',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-mac',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-tk',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-old',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-dynload',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/runipy-0.1.0-py2.7.egg',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/setuptools-3.6-py2.7.egg',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/PIL',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/IPython/extensions',
 '/Users/jimmy/.ipython']

And this is what os.environ['PATH'] returns:

'/Users/jimmy/dev/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin'

Both seem to place it in the correct order.

Grateful for all help.

Improve this question
4
  • 1
    Could it be that ! spawns a sub-shell that is different from the one you are normally using (like bash instead of zsh)? Commented Jul 10, 2014 at 9:17
  • @VolkerStampa I thought about that too, but the same thing happens in bash. The resulted IPython $PATH string isn't found in .bash_profile either. Commented Jul 10, 2014 at 9:41
  • What's os.environ['PATH'] inside ipython? That checks what the Python process sees, rather than what the shell spawned by ! sees. Commented Jul 10, 2014 at 15:13
  • @ThomasK Added the information as an edit. Commented Jul 10, 2014 at 16:47

2 Answers 2

Reset to default
10

To summarize and complement holdenweb's helpful answer, particularly with respect to macOS:

  • A subshell started from IPython with ! is a non-interactive non-login instance of the user's default shell - even if IPython was started from a different shell.
  • Behind the scenes, the subshell is started with path/to/default/shell -c ...
  • To see specifics, run !ps -p $$ && :

  • echo $SHELL always tells you the default shell - even when run from a different shell.

  • Initialization files sourced in non-interactive non-login shells:

    • zsh:: /etc/zshenv and ~/.zshenv
    • bash: a script pointed to in the $BASH_ENV variable, if defined.

  • As pointed out, shells load different/additional initialization files depending on whether:

    • the shell is a login shell or not
    • an interactive shell or not

    Note that a login shell can be interactive or not, and an interactive shell can be a login shell or not.

Thus, in the case at hand, potentially two additional initialization files were loaded in the interactive shell, explaining the difference in behavior between the interactive shell and the subshell created by IPython:

  • ~/.zprofile - if the shell is a login shell - which will be the case if zsh is the default shell (on macOS, all interactive instances of the default shell created in a terminal such as Terminal.app are login shells).
  • ~/.zshrc

Finally, on a related note, note that on macOS the default $PATH for NON-shell processes is:

/usr/bin:/bin:/usr/sbin:/sbin  # Note the absence of /usr/local/bin.

and only shells add /usr/local/bin (by default; extensible) to that, via system-wide initialization files (that call /usr/libexec/path_helper):

  • zsh:
    • /etc/zshenv
    • Note: takes effect for ALL zsh instances.
  • bash (also when invoked as sh), ksh:
    • /etc/profile
    • Note: takes effect for LOGIN shells only.

Upshot:

  • Non-shell apps see only the default $PATH (notably, without /usr/local/bin and other additions made via shell initialization files).
  • bash/sh and ksh non-interactive non-login shells not launched from a login shell also see only the default $PATH. This happens, for instance, when GUI command-line launchers such as Alfred create shell instances.
  • zsh is not affected, because /etc/zshenv is read by every zsh instance.
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Yes, nice answer
6

First of all, sys.path has little to do with this; it's just the list of locations in which the Python interpreter can look when importing modules, and does not determine where executable programs are found by the shell. Some elements of the PATH are seen in that list, though, because the interpreter uses the path of its executable to build certain entries on sys.path.

os.environ['PATH'] is, unsurprisingly, the same as the $PATH variable from the environment in which IPython is running. !echo $PATH prints out the $PATH variable from the subshell started by IPython to execute the shell escape (!).

One possible cause is that IPython is executing the shell commands using the OS standard shell which has not been tailored to set its path in the same way that your zsh has. You can confirm this by executing the IPython command !echo $SHELL. Since you confirmed this was not the case then the difference can be accounted for by the differences between a login shell and an interactive shell.

I don't know of a configuration item that will tell IPython to use another shell, but it's possible there is one. As a workaround, just make sure the other shell has a correctly configured path, or that interactive shells also see the environment you require.

See @mklement0's extremely authoritative answer for the whole, gory, messy details.

Improve this answer

5 Comments

Thanks for your answer, but running !echo $SHELL actually returns the same string as running the command outside of IPython, that is, /bin/zsh.
In which case it's likely the difference between running a login shell vs. an interactive shell. This document explains the difference well.
You are right; renaming my .zshrc file to .zshenv which runs for both interactive and login shells did the trick! Thank you!
@JimmyC: I think you meant to say that .zshenv is sourced for both interactive and non-interactive shells (i.e., scripts) - in other words: for all (zsh) shells. .zshrc, by contrast, is sourced by all interactive shells - whether login or not.
See this is why I didn't go into specifics in the answer. We should probably add a zsh tag if there is one. Done.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.