4

I am attempting something very similar to real time subprocess.Popen via stdout and PIPE

I, however, want to send input to the running process as well.

If I start a process in a separate thread using 

process = subprocess.Popen(cmd,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

I can send input using the terminal.

How would I send input from another source such as separate function not in the thread?

I cannot use Popen.communicate as the running process will never finish, as I am attempting real time interaction with the program.

Thanks in advance.

Here is my complete code, I am looking to have input sent to the subprocoess process when the send button is clicked.

from Tkinter import *`
from ttk import *`
import subprocess
from threading import Thread

class Example(Frame):

    def __init__(self, parent):
       Frame.__init__(self, parent)   

        self.parent = parent
        self.initUI()


    def initUI(self):    

        self.parent.title("Test Client")
        self.style = Style()
        self.style.theme_use("default")
        self.pack(fill=BOTH, expand=1)

        #Label, doesnt change
        lbl = Label(self, text="Client:")
        lbl.grid(row=0, column=1, sticky=W )

        #when output from client is shown
        global display
        display = Text(self,width=50,height=20)
        display.grid(row=1, column=1, sticky=E+W+N+S)

        #where user input is taken
        global prompt
        prompt = Entry(self,width=50)
        prompt.grid(row=3, column=1, sticky=E+W+N+S)

        #Button that will send input to client
        send = Button(self,text="Send",command=self.send)
        send.grid(row=3, column=2, sticky=N)
        get = Button(self,text="Get",command=self.get)
        get.grid(row=2, column=2, sticky=S)

    def get(self):
        print foo

    def send(self):
        sent = prompt.get()


def MyThread():
     global sent
     sent = 2
     cmd = ['nc', '-l', '-p', '50000']

     process = subprocess.Popen(cmd,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    while True:
        out = process.stdout.read(1)
        if out == '' and process.poll() != None:
            break
        if out != '':
            display.insert(INSERT, out)
            sys.stdout.write(out)
            sys.stdout.flush()

def main():
    root = Tk()
    root.geometry("500x410+300+300")
    app = Example(root)

    thread = Thread(target = MyThread, args=())
    thread.start()

    root.mainloop()

if __name__ == '__main__':
    main()
Improve this question
4
  • You should be able to use Popen.communicate() for that, have you actually tried it? What happen when you do? Commented Apr 18, 2013 at 19:06
  • 1
    @LieRyan: communicate waits for the child to finish and reads all of its output. Since "the running process will never finish", this means communicate will never return. So he can't use it. Commented Apr 18, 2013 at 19:09
  • 1
    you shouldn't probably call GUI from a background thread directly (it might or might not work intermittently). You could use a queue and root.after() to scheduler a call to GUI from the thread. Avoid using global unnecessarily, you could use instance variables instead e.g., self.prompt. You could use socket, select modules instead of running netcat. Commented Apr 18, 2013 at 20:55
  • 1
    @J.F.Sebastian: Good point about just using a socket directly instead of scripting netcat. (You'll probably still need a separate thread to service that socket… but better than 3-4 threads to service a subprocess… And a whole lot simpler. And it'll work on systems with BSD or Hobbit netcat instead of GNU netcat, or with no netcat at all. And so on.) Commented Apr 18, 2013 at 21:10

3 Answers 3

Reset to default
6

First, you obviously need to add stdin=subprocess.PIPE to the Popen constructor, and then you can process.stdin.write just as you process.stdout.read.

But obviously, just as read can block if there's no data yet, write can block if the child isn't reading.

And even beyond the obvious, it's actually very hard to get the details right for using PIPEs in both directions with Popen to an interactive program without blocking anywhere. If you really want to do it, look at the source for communicate to see how it works. (There are known bugs before 3.2, so if you're on 2.x, you may have to do some backporting.) You will have to implement the code yourself, and if you want it to be cross-platform, you're going to have to do the whole mess that communicate does internally (spawning reader and writer threads for the pipes, etc.), and of course add another thread to not block the main thread on each attempt to communicate, and some kind of mechanism to message the main thread when the child is ready, and so on.

Alternatively, you can look at the various "async subprocess" projects on PyPI. The simplest one I know of today is async_subprocess, which basically just gives you a communicate that you can use without blocking.

Or, if you can use twisted (or possibly other event-based networking frameworks), there are wrappers around subprocess that plug into its event loop. (If you can wait for Python 3.4, or use the work-in-progress tulip on 3.3, someone's built something similar around tulip that may make it into 3.4.) And twisted even knows how to plug into Tkinter, so you don't have to manually handle two separate event loops and communicate between them.

If you only care about modern POSIX systems (not Windows), you can make it simpler by just putting the pipes in non-blocking mode and writing your code as if you were dealing with sockets.

But the easiest solution is probably to use something like pexpect instead of trying to script it manually. (As J.F. Sebastian points out, pexpect is Unix-only, but you can use a wrapper around pexpect for Unix and winpexpect for Windows.)

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

I have looked at communicate and because its a blocking call it will not work with the commands I wish to run, pexpect is also unusable as I am not trying to script something but have an interactive program.
@MrS: I didn't say to use communicate, I said you have to look at the source to see all the things you will need to deal with in order to implement this yourself. Which is not going to be easy.
@MrS: Also, who told you that pexpect can't be used to script an interactive program? That's kind of the whole point of it.
there is also winpexpect module on Windows.
5

The select module in the standard library is made for these kind of situation:

process = subprocess.Popen(cmd,stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

while True:
   reads,writes,excs = select.select([process.stdout, process.stderr], [process.stdin], [], 1)
   for r in reads:
       out = r.read(1)
       display.insert(INSERT, out)
       sys.stdout.write(out)
       sys.stdout.flush()
   for w in writes:
       w.write('a')

You can pass a list of file objects or file descriptors to select() which will return those files that are have data ready for read/write or until an optional timeout.

The select module works on both Windows and Unix-like systems (Linux, Macs, etc).

Improve this answer

2 Comments

select is *nix only solution (only sockets can be used on Windows i.e., select doesn't work on pipes there). There should be process.poll() call somewhere to check whether the process is complete.
It works on most *nix platforms, but not all. And on some, it only works if you explicitly set the pipes to nonblocking first, which you aren't doing here.
0

A simple portable solution in your case with minimal code changes might be to create a writer thread that get items from a queue and writes them to the process' stdin and then put values into the queue whenever the button is pressed:

from subprocess import Popen, PIPE, STDOUT
from Queue import Queue

class Example(Frame):
    def __init__(self, parent, queue):
       # ...
       self.queue = queue
    # ...
    def send(self): # should be call on the button press
        self.queue.put(prompt.get())

def writer(input_queue, output): 
    for item in iter(input_queue.get, None): # get items until None found
        output.write(item)
    output.close()

def MyThread(queue):
    # ...
    #NOTE: you must set `stdin=PIPE` if you wan't to write to process.stdin
    process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    Thread(target=writer, args=[queue, process.stdin]).start()
    # ...

def main():
    # ...
    queue = Queue()
    app = Example(root, queue)
    Thread(target=MyThread, args=[queue]).start()
    # ...
    root.mainloop()
    queue.put(None) # no more input for the subprocess
Improve this answer

4 Comments

Might be useful for someone ; for me I have to add output.flush() after output.write() to make it work
@Raj bufsize=0 on Python 2; output=process.stdin and therefore output.flush() is useless in this case.
I am using Python 3.x. Any reason why I have to use flush()(believe buffer is getting filled up before pushed into the stream)...not sure if there is a better way
Both the question and my answer use Python 2. On Python 3, the default is different bufsize=-1 (block buffered mode) and output.flush() could be useful. There are other changes on Python 3 e.g., text vs. binary mode.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.