Classic problem with executing external commands in Java?

Question

Okay - This one is a bit of a problem for a lot of folks - Since I have yet to see an answer that works, I thought I would express the problem so someone who has figured it out can tell the rest of us.

The problem is that two out of three of the below work just fine -same code.

The instance for reader3 demonstrates the problem. Reader3 cannot read the result of the successful launch of an external file. Attempting to do any type of read (realine, etc.) on either the stdin or stderr InputStream blocks forever:

package Problems;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class RunningProblem {

public static class RunningReader implements Runnable {

    private Process proc;
    private String sName;

    private RunningReader(Process proc1, String sName) {
        this.proc = proc1;
        this.sName = sName;
    }

    public void run() {
        try {                
            // InputStreamReader in = new InputStreamReader(proc.getInputStream());
            // BufferedReader reader = new BufferedReader(in);

            InputStreamReader err = new InputStreamReader(proc.getErrorStream());
            BufferedReader reader = new BufferedReader(err);

            String line = reader.readLine();
            while (line != null) {
                System.out.println(sName + ": " + line);
                line = reader.readLine();
            }
            reader.close();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

public static void main(String[] args) {
    ExecutorService pool = Executors.newFixedThreadPool(3);
    try {
        Runtime rt = Runtime.getRuntime();

        Process proc1 = rt.exec("ps ax");
        RunningReader reader1 = new RunningReader(proc1, "reader1");

        Process proc2 = rt.exec("ls -l /");
        RunningReader reader2 = new RunningReader(proc2, "reader2");

        Process proc3 = rt.exec("/bin/tar");
        RunningReader reader3 = new RunningReader(proc3, "reader3");

        pool.execute(reader3);
        pool.execute(reader2);
        pool.execute(reader1);

    } catch (Exception ex) {
        System.err.println(ex.getMessage());
    } finally {
        pool.shutdown();
    }
    System.out.println("Launcher.main() Exited.");
}

}

Are you sure your /bin/tar prints to stderr (and not stdout ), and are you sure /bin/tar really stops when it's connected to a pipe ? — nos
– nos, Commented Mar 5, 2011 at 18:39
an application I'm working on is doing this daily, on hundreds of different machines... I can tell you one thing: there are way too many gotchas. We ended up making sure to never read nor write anything to any of the streams. In addition to that, the process we're spawning from Java is itself spawning another process, so that we can kill, from Java, the first process (after a while). We're then using temporary files for IPC. So basically we're wrapping any external process call in a "nohup ... &" call (that also takes care of redirecting stdout/stderr). — SyntaxT3rr0r
– SyntaxT3rr0r, Commented Mar 5, 2011 at 19:03
it kinda sucks, because we need to poll the temporary files but it's much easier that way. Just read what the various API trying to simplify Runtime.getRuntim().exec have to say about launching external processes from Java: it's a huge, messed up, fiasco. If you don't want to be as radical as we are, then at least use something like Apache commons exec and do read what they have to say about the pretty sad state of running external processes from Java. :) — SyntaxT3rr0r
– SyntaxT3rr0r, Commented Mar 5, 2011 at 19:06
Yea - Up until today, I thought a daemon process (written in C/C++ to be sure:) might have been the way-to-go... Now it seems that ProcessBuilder can work for me.... on Ubuntu... using Sun/Oracle JVM... Now that we have noted the problem, perhaps someone will be testing for it? (i.e. I wonder how this would work on my OSX Mac with the Apple JVM?) — SaintMagoo
– SaintMagoo, Commented Mar 6, 2011 at 12:57

Axel Knauf · Accepted Answer · 2011-03-05 18:51:27Z

1

You did not show the output your program yields, but I guess it might be the case, that ps ax and ls -l / do not yield output, but /bin/tar does. The reason is, that the first two commands produce output to stdout but not stderr, whereas the latter will produce output on stderr (since you are not giving valid parameters to tar) but not on stdout.

Here's the difference when running the commands in the shell:

[axe@gromp tmp]$ ps ax > ps-std.txt 2> ps-err.txt
[axe@gromp tmp]$ ls -l / > ls-std.txt 2> ls-err.txt
[axe@gromp tmp]$ /bin/tar > tar-std.txt 2> tar-err.txt
[axe@gromp tmp]$ ls -lrt
total 18
-rw-r--r-- 1 axe users    0 Mar  5 19:40 ps-err.txt
-rw-r--r-- 1 axe users 7191 Mar  5 19:40 ps-std.txt
-rw-r--r-- 1 axe users    0 Mar  5 19:40 ls-err.txt
-rw-r--r-- 1 axe users  937 Mar  5 19:40 ls-std.txt
-rw-r--r-- 1 axe users    0 Mar  5 19:41 tar-std.txt
-rw-r--r-- 1 axe users  142 Mar  5 19:41 tar-err.txt
[axe@gromp tmp]$

Using > to redirect the standard output and 2> to redirect the error output to different files, you can see that tar produces messages on stderr and the other two on stdout (file sizes for the other files are zero, there was no output).

May this be the case? What happens if you run, e. g. echo "Foo" instead of tar as third process?

answered Mar 5, 2011 at 18:51

Axel Knauf

1,7033 gold badges13 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

SaintMagoo Over a year ago

I was just looking to get a response from the process - problem was that any stream read attempt blocks forever - any output from reader3 would have been a success from the above, on my platform - (Thanks!:)

Luke Woodward · Accepted Answer · 2011-03-05 19:11:02Z

1

I ran your code on my system and it gave me the following output, before exiting normally:

reader3: /bin/tar: You must specify one of the `-Acdtrux' options
reader3: Try `/bin/tar --help' or `/bin/tar --usage' for more information.
Launcher.main() Exited.

There was no output from ps ax nor ls -l /, but running them from the shell confirms that they didn't write anything to standard error. On my system, your code happened to complete normally, but I can imagine situations where this doesn't happen. Do be aware that if a process generates a lot of output on its standard output, a buffer could fill up and that would cause the process to hang.

I'd recommend using a ProcessBuilder instead of Runtime.getRuntime().exec("..."). For one thing, it allows you to redirect the standard error stream into the standard output stream and then not have to worry about which of the two streams to read from.

answered Mar 5, 2011 at 19:11

Luke Woodward

65.3k16 gold badges94 silver badges108 bronze badges

2 Comments

nos Over a year ago

Yep, either redirect stderr/out, or spawn threads that read from them - otherwise you'll soon enough deadlock the processes you spawn and your own java threads.

SaintMagoo Over a year ago

Looks like this factors somewhere in on the Oracle JVM on Ubuntu: For me, the diff was 'internal' -v- 'external' - each flipping around upon choosing to work with the original, or the ProcessBuilder, approach. --Odd, wot?

SaintMagoo · Accepted Answer · 2011-03-06 12:12:41Z

Facinating (spockian eyebrow) - judging from the above comments, it looks as there are good reasons why so many folks have problems with this - our implementations each seem to work quite differently!

I am running Ubuntu. Interestingly, using ProcessBuilder inverted the problem... but at least none now seem to "block forever" when using it. -At least stderr and stdin were able to be read!

So for me, for now, the rule of thumb seems to be: Use the "old way" (Runtime.getRuntime().exec()) for command-shell ('internal') commands on Ubuntu (Oracle / Sun VM) -Use ProcessBuilder for external commands (like tar, etc.):

Process proc3 = new ProcessBuilder("/bin/tar").start();
RunningReader reader3 = new RunningReader(proc3, "reader3");

...

reader3: /bin/tar: You must specify one of the -Acdtrux' options reader3: Try/bin/tar --help' or `/bin/tar --usage' for more information.

--This is a pretty important operation for many of us ... It would be nice to put together a matrix on what-to-use on which-platform ....? (i.e. I wonder if OpenJDK would do better on Ubuntu?)

vivo · Accepted Answer · 2011-03-05 18:42:43Z

-1

You are probably starting the RunningReaders to late. You start three processes, and probably the first one will sometimes have finished before you start reading the Error-OutputStream in your RunningReader.

The RunningReaders must be started immediately after the process has started.

Under certain conditions even that might not be sufficient. Then you would have to create a wrapper script (for each OS) that captures the output and writes it to a file.

answered Mar 5, 2011 at 18:42

vivo

3332 silver badges8 bronze badges

2 Comments

nos Over a year ago

No, the running readers does not have to be started immetiately.

vivo Over a year ago

I am not completely sure about JDKs > 1.4, but up to 1.4 if you did not capture the output "fast" enough it could get lost on Windows & Solaris at least

Collectives™ on Stack Overflow

Classic problem with executing external commands in Java?

4 Answers 4

1 Comment

2 Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

2 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related