Python: Multithreading between Java subproccess and Python listener?

It's pretty hard to tell here, but I think what you're asking is how to:

• Launch a program in the background.
• Send it input, as if it came from a user on the console.
• Read its output that it tries to display to a user on the console.
• At the same time, run another thread that does other stuff.

The last one is pretty easy; in fact, you've mostly written it, you just need to add a thread1.start() somewhere.

The subprocess module lets you launch a program and control its input and output. It's easiest if you want to just feed in all the input at once, wait until it's done, then process all the output, but obviously that's not your case here, so it's a bit more involved:

minecraft = subprocess.Popen(['java', 'path/to/minecraft_server.jar', '-other', 'args],
                             stdin=subprocess.PIPE, 
                             stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

I'm merging stdout and stderr together into one pipe; if you want to read them separately, or send stderr to /dev/null, or whatever, see the docs; it's all pretty simple. While we're making assumptions here, I'm going to assume that minecraft_server uses a simple line-based protocol, where every command, every response, and every info message is exactly one line (that is, under 1K of text ending in a \n).

Now, to send it input, you just do this:

minecraft.stdin.write('Make me a sandwich\n')

Or, in Python 3.x:

minecraft.stdin.write(b'Make me a sandwich\n')

To read its output, you do this:

response = minecraft.stdout.readline()

That works just like a regular file. But note that it works like a binary file. In Python 2.x, the only difference is that newlines don't get automatically converted, but in Python 3.x, it means you can only write bytes (and compatible objects), not strs, and you will receive bytes back. There are good reasons for that, but if you want to get pipes that act like text files instead, see the universal_newlines (and possibly bufsize) arguments under Frequently Used Arguments and Popen Constructor.

Also, it works like a blocking file. With a regular file, this rarely matters, but with a pipe, it's quite possible that there will be data later, but there isn't data yet (because the server hasn't written it yet). So, if there is no output yet (or not a complete line's worth, since I used readline()), your thread just blocks, waiting until there is.

If you don't want that, you probably want to create another thread to service stdout. And its function can actually look pretty similar to what you've got:

def listener():
    for line in minecraft.stdout:
        if line.strip() == "Server can't keep up!":
            minecraft.stdin.write("Restart Universe\n")

Now that thread can block all day and there's no problem, because your other threads are still going.

Well, not quite no problem.

First it's going to be hard to cleanly shut down your program.

More seriously, the pipes between processes have a fixed size; if you don't service stdout fast enough, or the child doesn't service stdin fast enough, the pipe can block. And, the way I've written things, if the stdin pipe blocks, we'll be blocked forever in that stdin.write and won't get to the next read off stdout, so that can block too, and suddenly we're both waiting on each other forever.

You can solve this by having another thread to service stdout. The subprocess module itself includes an example, in the Popen._communicate function used by all the higher-level functions. (Make sure to look at Python 3.3 or later, because earlier versions had bugs.)

If you're in Python 3.4+ (or 3.3 with a backport off PyPI), you can instead use asyncio to rewrite your program around an event loop and handle the input and output the same way you'd write a reactor-based network server. That's what all the cool kids are doing in 2017, but back in late 2014 many people still thought it looked new and scary.

If all of this is sounding like a lot more work than you signed on for, you may want to consider using pexpect, which wraps up a lot of the tedious details, and makes some simplifying assumptions that are probably true in your case.