Run multiple bash lines in Python and separetely check their status and output

Question

I am trying to execute several lines of bash in Python 3 and check the status of each line separately.

I first tried to use gestatusoutput from subprocess, but each line is run in a separated process that does not communicate with the others (for the sake of simplicity, the given MWE consists of setting a variable, but what I intend to do in my actual code is more complex than that — and I know about os.environ for this very specific example):

from subprocess import getstatusoutput as cmd

stat, out = cmd("export TEST=1")
stat, out = cmd("echo $TEST")

will therefore returns:

>>> print(out)
(0, "")

I then tried the following:

cmdline = """export TEST=1
echo $TEST"""
stat, out = cmd(cmdline)

That works but forces me to parse the output, specially if I want to check the status of the first command (if echo works, the status returns by cmd is 0 whatever happens before), that is not very robust.

I saw some things using Popen (still from subprocess) but was unable to use it efficiently.

Any help would be appreciated!

If you want to execute multiple commands and process all their outputs, you need to parse them. There's no automatic way to know where each command's output ends. — Barmar
– Barmar, Commented Jan 24, 2019 at 8:28
You could put something like echo 'XXXXX' between each command, and use that as a delimiter. — Barmar
– Barmar, Commented Jan 24, 2019 at 8:29
It seems to me that there must be a way to work with the same process ID and therefore do things in a consistent context without resorting to parsing outputs. — MBR
– MBR, Commented Jan 24, 2019 at 8:34
Maybe consider using pexpect so you can "control" the other process and respond to its outputs... pexpect.readthedocs.io/en/stable/index.html — Mark Setchell
– Mark Setchell, Commented Jan 24, 2019 at 9:14
@MBR Are you sure you need to use shell commands at all? Most things you can do from the shell can also be done directly from Python. What are you trying to do that needs this complicated shell processing? — Barmar
– Barmar, Commented Jan 24, 2019 at 9:25

lawrence liu · Accepted Answer · 2019-01-24 08:59:52Z

0

To me, you are trying to share the environment variable between two process, which is not possible. It looks like this:

Process 1 python main.py                #TEST = "" 
    |Process 2-->"export TEST=1"        #Change Process2 env variable TEST to '1'
    |Process 3-->"echo $TEST"           #Print Process3 env variable TEST (get from process 1)

You can use os.environ[] to change the current environment first (Process 1 variable),Later on use the variable after fork.

Something like this

import os
import subprocess
import sys

os.environ['TEST'] = '1'
out = subprocess.check_call('echo $TEST',shell = True)

answered Jan 24, 2019 at 8:59

lawrence liu

11 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

MBR Over a year ago

Thanks for your answer, but it does not answer my need as I gave this example for the sake of simplicity. What I intend to do in my actual code is more complex than just setting a variable.

MBR · Accepted Answer · 2019-03-08 10:45:18Z

I resulted doing the following:

create a launch command wrapping subprocess.Popen to launch my bash commands, that in addition allows me either to retrieve the current environment or to pass a custom environment
create a get_env to parse the return from the previous command and get a dict of the environment

`launch` wrapper

import os
import subprocess as sp

def launch(cmd_, env=os.environ, get_env=False):
    if get_env: cmd_ += " && printenv"
    load = sp.Popen(cmd_, shell=True, stdout=sp.PIPE, stderr=sp.PIPE, env=env)
    out  = load.communicate()
    err  = load.returncode
    return(err, out)

Retrieve the environment

def get_env(out, encoding='utf-8'):
    lout = str(out[0], encoding).split('\n')
    new_env = {}
    for line in lout:
        if len(line.split('=')) <= 1:
            pass
        else:
            k = line.split("=")[0]
            v = "=".join(line.split("=")[1:])
            new_env[k] = v
    return new_env

(This is a simple version, it may be more complicated if you have things like functions in your environment — it happens.)

Results:

I can use it as follow:

err, out = launch("export TEST=1", get_env=True)
if not err: new_env = get_env(out)
err, out = launch("echo $TEST", env=new_env)

and therefore:

>>> print(str(out[0], encoding='utf-8'))
1

Collectives™ on Stack Overflow

Run multiple bash lines in Python and separetely check their status and output

2 Answers 2

1 Comment

`launch` wrapper

Retrieve the environment

Results:

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

launch wrapper

Retrieve the environment

Results:

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related

`launch` wrapper