I'm telling jupyter to execute a python script:
!python build_database.py
When executed from the terminal, the python script prints the progress during execution. However, in the jupyter notebook, I get all output printed as a list of strings, AFTER the execution. Is there a way to see the output live?
It looks like it is not possible out of the box. The output handling of shell commands is buried deep in ipython internals.
One of solutions i would recommend is to create custom magic method based on code below.
Check this answer
Based on it i created a simple magic method that you can use:
from subprocess import Popen, PIPE, STDOUT
from IPython.core.magic import register_line_magic
@register_line_magic
def runrealcmd(command):
process = Popen(command, stdout=PIPE, shell=True, stderr=STDOUT, bufsize=1, close_fds=True)
for line in iter(process.stdout.readline, b''):
print(line.rstrip().decode('utf-8'))
process.stdout.close()
process.wait()
Or Popen can be used as context manager. So the code will be a bit more readable and reliable. See docs
Popen objects are supported as context managers via the with statement: on exit, standard file descriptors are closed, and the process is waited for.
from subprocess import Popen, PIPE, STDOUT
from IPython.core.magic import register_line_magic
@register_line_magic
def runrealcmd(command):
with Popen(
command, stdout=PIPE, shell=True, stderr=STDOUT, bufsize=1, close_fds=True
) as process:
for line in iter(process.stdout.readline, b""):
print(line.rstrip().decode("utf-8"))
Usage:
%runrealcmd ping -c10 www.google.com
Above code probably could be written better but for your needs it should be perfectly fine.
Popen as context manager. I've added example in my original answer.I refer to this answer: How to show real time process command line in Jupyter Notebook?
Just execute your python script as:
%run build_database.py
%runand not!python. In this example, that would be%run build_database.py. It often gives a different, better experience depending on your system. Plus, it has more options, see here.