Is there a way to both stream and capture output from subprocess.run in Python 3?

Question

If you do

subprocess.run(["echo hi && sleep 60"], shell=True, capture_output=False, timeout=5)

you will see 'hi' printed to the terminal and then an exception will be thrown due to the timeout.

But if you do

ret = subprocess.run(["echo hi && sleep 6"], capture_output=True, timeout=2, shell=True)

You won't see any output, and the timeout exception will (as far as I can tell) prevent you from getting the output that was produced. (The shell=True is not important for this example, I just needed a convenient way to make it timeout.)

This is causing issues where I (a) need to get the output of the command but (b) sometimes am inadvertently running commands that hang waiting for user input. I'd really like to be able to capture any output produced despite a Timeout Exception being thrown or stream the output to stdout/stderr while also getting a copy of it.

Can you clarify what is supposed to be the relevant difference between the two samples? (1) that you assign a return value (2) that sleep was changed from 60 to 6 (3) that capture_output changed from False to True (4) that timeout was changed from 5 to 2 (5) that the order of the arguments has been changed around? It would be nice to see two examples where only one thing changes. — mkrieger1
– mkrieger1, Commented Mar 13, 2020 at 19:23

ShadowRanger · Accepted Answer · 2020-03-13 19:25:03Z

The TimeoutExpired exception itself includes any captured output as attributes (stdout and stderr), so:

try:
    ret = subprocess.run("commands go here && sleep 6", capture_output=True, timeout=2, shell=True)
except subprocess.TimeoutExpired as e:
    print(e.stdout)
else:
    print(ret.stdout)

will work when the command in question actually writes to the pipe prior to the timeout.

But if the subprocess was killed before completion, while it still had data in user-mode buffers (it was block buffering, or line buffering and never got around to writing a newline or whatever), it would never have been written to the actual pipe, so you won't see it.

Unbuffering the child process is done in many different ways, so I can't give specific info on how to do it in your real case (echo is obviously just a placeholder), the stdbuf is often useful when the child executable can't be modified to make it line-buffered or unbuffered.

Collectives™ on Stack Overflow

Is there a way to both stream and capture output from subprocess.run in Python 3?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related