printing stdout in realtime from a subprocess that requires stdin

Question

This is a follow up to this question, but if I want to pass an argument to stdin to subprocess, how can I get the output in real time? This is what I currently have; I also tried replacing Popen with call from the subprocess module and this just leads to the script hanging.

from subprocess import Popen, PIPE, STDOUT
cmd = 'rsync --rsh=ssh -rv --files-from=- thisdir/ servername:folder/'
p = Popen(cmd.split(), stdout=PIPE, stdin=PIPE, stderr=STDOUT)
subfolders = '\n'.join(['subfolder1','subfolder2'])
output = p.communicate(input=subfolders)[0]
print output

In the former question where I did not have to pass stdin I was suggested to use p.stdout.readline, there there is no room there to pipe anything to stdin.

Addendum: This works for the transfer, but I see the output only at the end and I would like to see the details of the transfer while it's happening.

if all you want is to see the output then just drop stdout=PIPE. — jfs
– jfs, Commented Mar 4, 2014 at 14:47
just use this link https://stackoverflow.com/a/75175057/12780274 is very simple — henrry
– henrry, Commented Jan 19, 2023 at 16:06
Does this answer your question? Wrap subprocess' stdout/stderr — Victor Sergienko
– Victor Sergienko, Commented Feb 20, 2024 at 21:33

Alp · Accepted Answer · 2014-03-05 15:17:36Z

39

In order to grab stdout from the subprocess in real time you need to decide exactly what behavior you want; specifically, you need to decide whether you want to deal with the output line-by-line or character-by-character, and whether you want to block while waiting for output or be able to do something else while waiting.

It looks like it will probably suffice for your case to read the output in line-buffered fashion, blocking until each complete line comes in, which means the convenience functions provided by subprocess are good enough:

p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE)
# Grab stdout line by line as it becomes available.  This will loop until 
# p terminates.
while p.poll() is None:
    l = p.stdout.readline() # This blocks until it receives a newline.
    print l
# When the subprocess terminates there might be unconsumed output 
# that still needs to be processed.
print p.stdout.read()

If you need to write to the stdin of the process, just use another pipe:

p = subprocess.Popen(some_cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
# Send input to p.
p.stdin.write("some input\n")
p.stdin.flush()
# Now start grabbing output.
while p.poll() is None:
    l = p.stdout.readline()
    print l
print p.stdout.read()

Pace the other answer, there's no need to indirect through a file in order to pass input to the subprocess.

edited Mar 5, 2014 at 15:17

answered Jul 1, 2013 at 20:25

Alp

2,84621 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Joran Beasley Over a year ago

i dunno I couldnt get sys.stdin.read() to return anything in the called file with python2.6 with this method ... thats why I didnt put that as an answer ... but maybe i broke something.

Alp Over a year ago

When called with no argument read() won't return until it encounters EOF (it's trying to read the entirety of the file object it's called on, and it doesn't know it's consumed all of the file until it reaches EOF). That's rarely the behavior one wants when reading input. You should either pass a specified number of bytes as argument to read() (in which case the read() call will return after encountering that number of bytes or EOF, whichever comes first) or use readline() (which will return after encountering a newline).

jfs Over a year ago

your code will deadlock if subprocess generates enough output on stderr. Don't use stderr=PIPE unless you read from p.stderr later.

Mark E. Hamilton Over a year ago

When I tried this I got extra whitespace, because the readline() method returns the newline, and print() adds a newline. I changed the print statements to sys.stdout.write() and got the output I expected.

Tagar Over a year ago

Is there is a way to read either stdout and stderr at the same time? Meaning, like if you would run a shell command, it would print whichever comes first. I'd like to have same behavior.. Since p.stdout.readline() is blocking, not sure how to achieve this. Thanks

|

Joran Beasley · Accepted Answer · 2013-07-01 21:56:20Z

3

something like this I think

from subprocess import Popen, PIPE, STDOUT

p = Popen('c:/python26/python printingTest.py', stdout = PIPE, 
        stderr = PIPE)
for line in iter(p.stdout.readline, ''):
    print line
p.stdout.close()

using an iterator will return live results basically ..

in order to send input to stdin you would need something like

other_input = "some extra input stuff"
with open("to_input.txt","w") as f:
   f.write(other_input)
p = Popen('c:/python26/python printingTest.py < some_input_redirection_thing', 
         stdin = open("to_input.txt"),
         stdout = PIPE, 
         stderr = PIPE)

this would be similar to the linux shell command of

%prompt%> some_file.o < cat to_input.txt

see alps answer for better passing to stdin

edited Jul 1, 2013 at 21:56

answered Jul 1, 2013 at 19:21

Joran Beasley

114k13 gold badges167 silver badges187 bronze badges

2 Comments

hatmatrix Over a year ago

Thanks, this was an answer given in my previous question where I didn't need stdin, but I do not know how to pass to it the 'input' argument as I would to p.communicate

Joran Beasley Over a year ago

see edits ... you would need to put the input string in a file and pass an open filehandler to the stdin argument of the constructor

jfs · Accepted Answer · 2014-03-04 14:46:24Z

2

If you pass all your input before starting reading the output and if by "real-time" you mean whenever the subprocess flushes its stdout buffer:

from subprocess import Popen, PIPE, STDOUT

cmd = 'rsync --rsh=ssh -rv --files-from=- thisdir/ servername:folder/'
p = Popen(cmd.split(), stdout=PIPE, stdin=PIPE, stderr=STDOUT, bufsize=1)
subfolders = '\n'.join(['subfolder1','subfolder2'])
p.stdin.write(subfolders)
p.stdin.close() # eof
for line in iter(p.stdout.readline, ''):
    print line, # do something with the output here
p.stdout.close()
rc = p.wait()

answered Mar 4, 2014 at 14:46

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Comments

cards · Accepted Answer · 2025-04-17 21:55:57Z

To print the stdout in realtime pass a None value to the stdout parameter instead of a PIPE.

Here a small example of usage with input given as a "plain" str by passing text=True instead of a text in binary form.

import subprocess as sb

# some program
stdins = "sample input 1", "sample input 2"
cmd = "python3", "path_to_program.py"

with sb.Popen(cmd, stdout=None, stdin=sb.PIPE, text=True) as proc:
    outs, _ = proc.communicate('\n'.join(stdins))
print(f"PID {proc.pid} rc {proc.returncode}")

Notice also that the doc recomend to use Popen.communicate over explicit manipulations of the standard stream decsriptors stdin, stdout, stderr:

Warning: Use communicate rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

This answer is based on the first comment of the OP, by jfs, i.e. simply use the default value of stdout. At present time, I struggled with the other answers so I wrote this one.

Daanturo · Accepted Answer · 2025-08-29 06:34:20Z

0

If Python 3, try https://docs.python.org/3/library/pty.html#pty.spawn:

import pty

pty.spawn(
    ["rsync", "--rsh=ssh", "-rv", "--files-from=-", "thisdir/", "servername:folder/"]
)

answered Aug 29 at 6:34

Daanturo

2303 silver badges7 bronze badges

Collectives™ on Stack Overflow

printing stdout in realtime from a subprocess that requires stdin

5 Answers 5

9 Comments

2 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

9 Comments

2 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related