Capturing subprocess in console and python

Question

I'm trying to execute some bash command using python. I want to display command's live output to user as well as capture it.
A sample example is like

import subporcess

# This will store output in result but print nothing to terminal
result = subprocess.run(['ls', '-lh'], check=True, universal_newlines=True, stdout=subprocess.PIPE)
print(result.stdout) # STD OUTPUT
# This will print everything to terminal result will be empty
result = subprocess.run(['ls', '-lh'], check=True, universal_newlines=True)
print(result.stdout) # OUTPUT = None

instead of .run you can use .check_output. Have a look at the documentation — Ma0
– Ma0, Commented Jun 21, 2021 at 13:26
If you will go through the function arguments you'll see that there is not stdout option. i.e checkout will only return output to python but print nothing on the console. — Pranjal Doshi
– Pranjal Doshi, Commented Jun 21, 2021 at 13:39
Yes.. But The process I am trying to run takes around 30 min till then the user will think that the program is dead. — Pranjal Doshi
– Pranjal Doshi, Commented Jun 21, 2021 at 13:45
in that case you can subclass subprocess.PIPE and make it write the output in two places at the same time. — Ma0
– Ma0, Commented Jun 21, 2021 at 13:49

Simon Bowly · Accepted Answer · 2021-06-21 13:55:46Z

2

Here is one possibility which will gather output lines from a long-running process, write them to terminal as it goes, and return them all when the process exits.

It returns a list of output lines, rather than a full block of text that check_output or run would return, but that's easily changed. Perhaps an IO buffer might be more efficient depending on how much output you're expecting.

import subprocess
import sys

def capture_and_echo_stdout(cmd):
    """ Start a subprocess and write its stdout to stdout of this process.     
    Capture and return stdout lines as a list. """
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    stdout_lines = []
    for line in proc.stdout:
        sys.stdout.write(line.decode())
        stdout_lines.append(line)
    proc.communicate()
    # Roughly equivalent to check=True
    if proc.returncode:
        raise subprocess.CalledProcessError(proc.returncode, cmd)
    return stdout_lines

There are a few similar options in this answer (although the focus there is more on writing to multiple files like unix tee): How to replicate tee behavior in Python when using subprocess?

answered Jun 21, 2021 at 13:55

Simon Bowly

1,0935 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Pranjal Doshi Over a year ago

Thanks.. it is one of the option that I considered. But run is pretty stable and covers some corner cases which I won't be able to handle while using Popen. But thanks for great answer.

Collectives™ on Stack Overflow

Capturing subprocess in console and python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related