Start background process in python via subprocess and write output to file

Question

Dear stackoverflow users,

I'm looking for a solution for a probably quite easy problem. I want to automate some quantum chemical calculations and ran into a small problem.

Normally you start your quantum chemical programm (in my case it's called orca) with your input file (*.inp) on a remote server as a background process and pipe the output into an outputfile (*.out) via

nohup orca H2.inp >& H2.out &

or something similar.

Now I wanted to use a python script (with some templating) to write the input file automatically. At the end the script should start the calculation in a way that I could log out of the server without stopping orca. I tried that with

subprocess.run(["orca", input_file], stdout=output_file)

but so far it did not work. How do I "emulate" the command given at the top with the subprocess module?

Regards

Update I have one file that is called H2.xyz. The script reads and splits the filename by the point and creates an input file name H2.inp and the output should be written into the file H2.out.

Update 2 The input file is derived from the *xyz file via

xyzfile = str(sys.argv[1])
input_file = xyzfile.split(".")[0] + ".inp"
output_file = xyzfile.split(".")[0] + ".out"

and is created within the script via templating. In the end I want to run the script in the following way:

python3 script.py H2_0_1.xyz

"If the shell is invoked explicitly, via shell=True, it is the application’s responsibility to ensure that all whitespace and metacharacters are quoted appropriately to avoid shell injection vulnerabilities" from docs.python.org/3/library/… — Sharku
– Sharku, Commented Aug 30, 2018 at 11:35
is output_file the path to a file (I don't think that will work) or a file-like object (like what you get from using the open(output_file, 'w') method? — Daren Thomas
– Daren Thomas, Commented Aug 30, 2018 at 12:01
So far I did not get it to work, so I'm open to any solution. — JakobJakobson13
– JakobJakobson13, Commented Aug 30, 2018 at 13:40

Adelina · Accepted Answer · 2018-08-30 11:58:11Z

1

Why not simply:

subprocess.Popen(f'orca {input_file} >& {output_file}',
                 shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)

More info: Run Process and Don't Wait

edited Aug 30, 2018 at 11:58

answered Aug 30, 2018 at 11:52

Adelina

12k1 gold badge44 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sharku Over a year ago

like I commented in the question shell=True is not secure, because of injection vulnerabilities. In my solution Python is taking care of that. But yes I might be over killing it xDDD

Sven-Eric Krüger · Accepted Answer · 2018-08-30 13:52:15Z

1

For me (Windows, Python 2.7) the method call works very fine like this:

with open('H2.out', 'a') as out :
    subprocess.call(['orca', infile], stdout=out,
                                      stderr=out,
                                      shell=True)   # Yes, I know. But It's Windows.

On Linux you maybe do not need shell=True for a list of arguments.

answered Aug 30, 2018 at 13:52

Sven-Eric Krüger

1,35712 silver badges19 bronze badges

Comments

Simon K. · Accepted Answer · 2018-08-30 12:06:36Z

0

Is the usage of subprocess important? If not, you could use os.system.

The Python call would get really short, in your case

os.system("nohup orca H2.inp >& H2.out &")

should do the trick.

answered Aug 30, 2018 at 12:06

Simon K.

665 bronze badges

Comments

Sharku · Accepted Answer · 2018-08-30 13:00:27Z

I had the same problem not long ago. Here is my solution:

commandLineCode = "nohup orca H2.inp >& H2.out &"
try:
    proc = subprocess.Popen(commandLineCode,
                            stdin=subprocess.PIPE,
                            stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE,
                            cwd = workingDir)

except OSError:
    print("Windows Error occured")
    print(traceback.format_exc())

timeoutInSeconds = 100
try:
    outs, errs = proc.communicate(timeout = timeoutInSeconds)
except subprocess.TimeoutExpired:
    print("timeout")
    proc.kill()
    outs, errs = proc.communicate()

stdoutDecode = outs.decode("utf-8")
stderrDecode = errs.decode("utf-8")

for line in stdoutDecode.splitlines():
    # write line to outputFile
if stderrDecode:
    for line in stderrDecode.splitlines():
        # write line to error log

The OSError exception is pretty important since you never now what your OS might do wrong.

For more on the communicate() command which actually starts the process read: https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate

Collectives™ on Stack Overflow

Start background process in python via subprocess and write output to file

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related