How to properly interact with a process using subprocess module

Question

I'm having problems redirecting stdio of another program using subprocess module. Just reading from stdout results in hanging, and Popen.communicate() works but it closes pipes after reading/writing. What's the easiest way to implement this?

I was playing around with this on windows:

import subprocess
proc = subprocess.Popen('python -c "while True: print \'Hi %s!\' % raw_input()"',
                        shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT)
while True:
    proc.stdin.write('world\n')
    proc_read = proc.stdout.readline()
    if proc_read:
        print proc_read

Duplicate: stackoverflow.com/questions/163542/…, stackoverflow.com/questions/295459/… — S.Lott
– S.Lott, Commented Jan 14, 2009 at 14:24
-u flag would solve it for a Python subprocess. There are also pexpect, pty modules and unbuffer, stdbuf, script utilities that can help to fix the block-buffering issue. — jfs
– jfs, Commented May 22, 2013 at 19:33

Aaron Digulla · Accepted Answer · 2009-01-14 14:16:38Z

21

Doesn't fit 100% to your example but helps to understand the underlying issue: Process P starts child C. Child C writes something to its stdout. stdout of C is a pipe which has a 4096 character buffer and the output is shorter than that. Now, C waits for some input. For C, everything is fine.

P waits for the output which will never come because the OS sees no reason to flush the output buffer of C (with so little data in it). Since P never gets the output of C, it will never write anything to C, so C hangs waiting for the input from P.

Fix: Use flush after every write to a pipe forcing the OS to send the data now.

In your case, adding proc.stdin.flush() in the main while loop and a sys.stdout.flush() in the child loop after the print should fix your problem.

You should also consider moving the code which reads from the other process into a thread. The idea here is that you can never know when the data will arrive and using a thread helps you to understand these issues while you write the code which processes the results.

At this place, I wanted to show you the new Python 2.6 documentation but it doesn't explain the flush issue, either :( Oh well ...

answered Jan 14, 2009 at 14:16

Aaron Digulla

330k111 gold badges626 silver badges840 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

jfs Over a year ago

Good explanation. Here's more from pexpect docs: Q: Why not just use a pipe (popen())?

jfs Over a year ago

Pexpect docs has moved: here's new link

Collectives™ on Stack Overflow

How to properly interact with a process using subprocess module

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related