Real Time updating executable subprocess console output in Python

Question

I have been reading this question, and this one and noted that the solutions proposed there did not do the trick I'm trying to pull off. The idea is similar to this question but I'm giving a better example here.

I need to build a GUI in Python 3.7 to monitor and control the execution of a legacy software written in C. Right now, I'm trying to simply call a test executable from Python and have it printed on a GUI, but for now it would suffice to print on the same console as python. The problem is that the executable takes very long to fully execute, but prints console messages in the meantime, I need these messages to be read promptly by my GUI.

Here is a working example:

in Python 3.7:

import sys
from threading import Thread
import time
import subprocess

class HandleTarget(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        subprocess.Popen(["hello.exe"], stdout=sys.stdout, bufsize=1)

class Printer(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        for i in range(1, 15):
            sys.stdout.write("Hi back ("+str(i)+")!\n")
            time.sleep(1.0)

thread_1 = HandleTarget()
thread_2 = Printer()

# Launching threads:
thread_1.start()
thread_2.start()

Now, the example executable ("hello.exe") can be build in C like this:

#include <stdio.h>
#include <time.h>

int main() {
    struct timespec time;
    double tic = 0;
    double toc = 0;

    for( int ii = 0; ii < 3 ; ii++){
        clock_gettime(CLOCK_MONOTONIC, &time);
        tic = time.tv_sec;
        toc = tic;

        while(toc-tic<3) {
        clock_gettime(CLOCK_MONOTONIC, &time);
        toc    = time.tv_sec;
        }
        printf("Hello #%d \n",ii);
    }
    return(0); 
}

Ideally, I need the messages "Hello" and "Hi back" to be interleaved. But if I run the python script above, I get:

Hi back (1)!
Hi back (2)!
Hi back (3)!
Hi back (4)!
Hi back (5)!
Hi back (6)!
Hi back (7)!
Hi back (8)!
Hi back (9)!
Hello #0 
Hello #1 
Hello #2 
Hi back (10)!
Hi back (11)!
Hi back (12)!
Hi back (13)!
Hi back (14)!

Apparently, the output from the executable is only printed when the execution is finished. This means that if the legacy executable was running, only upon being finished anything would show up.

Edit: I do not have hard real time constraints on this, if printing actually takes a few minutes, that's fine, the problem is that if a process needs to run for days, then every few hours (preferably minutes) the GUI needs to be updated with the console prints from the executable.

output from executable is printed when there is "\n". But Python doesn't run threads at the same time - it runs one of them for awhile, next it runs second of them for awhile, next it runs first for awhile, etc. Sometimes on thread may run longer than another and you can get all text from one thread at once. I would use stdout=PIPE to read line from subprocess (when line has "\n") and display it in GUI and then wait for next line from subprocess. — furas
– furas, Commented Apr 4, 2019 at 23:15
By inverted, do you mean, instead of the "hi back" it will print "hello", and vice versa? — xilpex
– xilpex, Commented Apr 7, 2019 at 22:36
@Xilpex : I actually wrote "intervened", which means that the hello messages should not all be together, but rather there should be one hi back between two hello. — Mefitico
– Mefitico, Commented Apr 7, 2019 at 23:34
@Massimo According to Dictionary.com Intervene "2.to occur or be between two things. 3 to occur or happen between other events or periods". But I agree "interleaved" would be clearer. — Mefitico
– Mefitico, Commented Jun 13, 2019 at 12:05

gbajson · Accepted Answer · 2019-04-10 14:00:34Z

4

+150

Consider reading Popen.stdout:

https://docs.python.org/3/library/subprocess.html?highlight=subprocess#subprocess.Popen.stdout

Here is the output of the updated program:

$ python test.py
Hi back (1)!
Hi back (2)!
Hi back (3)!
Hello #0
Hi back (4)!
Hi back (5)!
Hi back (6)!
Hello #1
Hi back (7)!
Hi back (8)!
Hi back (9)!
Hello #2
Hi back (10)!
Hi back (11)!
Hi back (12)!
Hi back (13)!
Hi back (14)!

The updated program:

import sys
from threading import Thread
import time
import subprocess

class HandleTarget(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        proc = subprocess.Popen(["hello.exe"], stdout=subprocess.PIPE)
        for line in proc.stdout:
            print(line.strip().decode('utf-8'))


class Printer(Thread):
    def __init__(self):
        Thread.__init__(self)

    def run(self):
        for i in range(1, 15):
            sys.stdout.write("Hi back ("+str(i)+")!\n")
            time.sleep(1.0)

thread_1 = HandleTarget()
thread_2 = Printer()

# Launching threads:
thread_1.start()
thread_2.start()

EDIT:

I also modified C program to flush stdout after printing, buffered stdout seems to be the root cause of the problem:

    printf("Hello #%d \n",ii);
    fflush(stdout);

edited Apr 10, 2019 at 14:00

answered Apr 10, 2019 at 13:05

gbajson

1,83019 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mefitico Over a year ago

On the bright side, this worked here as well. But as mentioned in the question, there is little I'm allowed to do with the legacy software, the hello function is meant to be a minimal working example. I might try to redefine printf to always perform an fflush and preload a custom lib, or something. Nonetheless, if no better answer comes about, you are eligible to the bounty.

Mefitico Over a year ago

I checked that it also works to add setvbuf(stdout, NULL, _IONBF, 0); (prevent buffering altogether). This might be easier on my requirements, you might consider testing/adding to your answer.

gbajson Over a year ago

@Mefitico I recompiled C program in Linux and it prints "Hello #%d \n" every second. The same source compiled on Windows flushes buffer at once when program ends. Therefore I think it's rather a problem with the buffers in Windows terminal, than Python.

Collectives™ on Stack Overflow

Real Time updating executable subprocess console output in Python

1 Answer 1

EDIT:

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

EDIT:

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related