Can I execute a shell command that requires input in ipython and/or an ipython notebook?
When I execute such a command, I see it's prompt, but no apparent way to provide it with input from my keyboard.
An example could be an rsync command to a remote server (thus requiring a password). There are no doubt dangers security-wise here - these are somewhat reduced in my case as I'm running on localhost.
Reposting as an answer, with a bit more detail:
No, it's a long standing issue that's really difficult to resolve: github.com/ipython/ipython/issues/514
The issue is roughly that our architecture can't tell when a process is waiting for input, so it doesn't know to prompt the user for input.
You may be able to use pexpect and raw_input to simulate this for specific programs. I.e. if you say what the prompt looks like, it spots that in the output, and asks the user for input to send back to the process. E.g. for Python:
import pexpect
p = pexpect.spawn('python')
while True:
try:
p.expect('\n>>> ')
print(p.before)
p.sendline(raw_input('>>> '))
except pexpect.EOF:
break
I've just given this a brief test - it works, but it's fairly rough. The concept could be improved.
Sadly, no.
I couldn't find documentation on this, so I went source-diving. The ipython code that actually performs that transformation is https://github.com/ipython/ipython/blob/master/IPython/core/inputtransformer.py#L208 , specifically:
def _tr_system(line_info):
"Translate lines escaped with: !"
cmd = line_info.line.lstrip().lstrip(ESC_SHELL)
return '%sget_ipython().system(%r)' % (line_info.pre, cmd)
which, in other words, invokes the underlying shell with everything following the !.
ipython is probably not what you want -- check out this answer for a Python alternate to include in scripts.
Was just looking for this and my wee face dropped when I saw it was a bit of an issue. Thought I would just post my solution in case it is usefull to anyone else.
Basically I was looking for a way to send sudo commands through the notebook, probably not very wise but I needed it for what I was doing. And i couldn't get a prompt for the password. So decided to use a x-terminal and sending the command through to the terminal. You don't get any feed back but may be due to not hooking to the IO on the way back. Here is what i used in the notebook:
In [1] !xterm -e sudo mount -o loop system.img /system/
I'm using linux but i would expect !cmd for windows might do the trick too
Many programs that require a password provide a variety of ways to prompt the user for it so that jupyter doesn't have to do it for you. The sudo command has a -A option and the SUDO_ASKPASS environmental variable. I've used it like this to use sudo to get permissions for the /var/log/btmp.1 file:
In [1]: !SUDO_ASKPASS=/usr/bin/ssh-askpass sudo -A lastb -a -f /var/log/btmp.1 | head
Similarly, for your case, ssh has an SSH_ASKPASS environmental variable.
Note that for headless operation of ssh/rsync, you can avoid authentication prompts from a notebook entirely by directly setting up an ssh agent (if it isn't running already) and referring to it with your SSH_AUTH_SOCK variable, like ssh-add does.
Or you can use a different program via SSH_ASKPASS which handles authentication in a custom way. See questions like How to automate SSH login with password? - Server Fault for more details, but be careful to not compromise your security especially when trying to automate connections from one server to another.
ssh-askpass is a gui program, no? Ok then if you're running on a desktop machine, not so handy if ipython/jupyter is running on a headless server.