Reading stdout of non-python subprocess continuously

Question

I have an executable that I need to run which outputs messages to the stdout while running, however when using subprocess, it seems to wait for the script to finish completely before sending the messages.

Note: Using Windows 10

The executable that I will need to run is not a Python process, so it can not be fixed by adding sys.stdout.flush(), python -u ... etc.

main.py

import subprocess

popen = subprocess.Popen(["my_process"], stdout=subprocess.PIPE, universal_newlines=True)
for line in popen.stdout: 
    print(line)

Anyone have any ideas how I can fix this? I am planning to implement this in a PyQt gui using QProcess, though the problem is also present in that implementation.

Why you cant use the 2 fixes you mentioned ? Additionally you could set -u using env variable — Maurice Meyer
– Maurice Meyer, Commented Aug 14, 2024 at 9:31
I guess the actual executable in the subprocess isn't a Python script? Have you checked (by directly calling the executable in the console) if it really outputs during (and not after) execution? — Michael Butscher
– Michael Butscher, Commented Aug 14, 2024 at 10:32
Yes the reason I can't use -u is because its not a python file but an executable. When I run the executable through cmd it does output information continuously. I did however notice that when I replace the print in the loop with "print(f"{i}" * 10000)", it does output it directly, so it appears to be a buffer problem? Though when I change the buffer of Popen to 0 it still doesn't work. — bgen
– bgen, Commented Aug 14, 2024 at 12:02
No, that was just to demonstrate the same effect. Sorry if that wasn't clear. The application I actually have to run is a compiled executable which I have no way to edit. — bgen
– bgen, Commented Aug 14, 2024 at 12:31

ticktalk · Accepted Answer · 2024-08-19 15:30:51Z

0

if you have WSL ... believe it has the stdbuf utility (at least thats what ubuntu alleges), it may also have the unbuffer utility mentioned (no experience of using that btw)

bgen.py
import subprocess
import sys
import datetime

cmd = [ "/home/tester/someprocess" ]

if 'stdbuf' in sys.argv:
    cmd = [ "/usr/bin/stdbuf", "-o0", "/home/tester/someprocess" ]

print(f'{datetime.datetime.now()} :: running: {cmd}' )

popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
for line in popen.stdout: 
    print(f'{datetime.datetime.now()} :: got:{line.rstrip()}')

someprocess.c
include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv)
{
    for( int iter=0; iter <= 5; iter++ ){
        printf( "iteration %d\n", iter );
        sleep(1);
    }
}

# buffered 
python bgen.py
2024-08-19 16:18:44.366847 :: running: ['/home/tester/someprocess']
2024-08-19 16:18:50.368800 :: got:iteration 0
2024-08-19 16:18:50.368871 :: got:iteration 1
2024-08-19 16:18:50.368908 :: got:iteration 2
2024-08-19 16:18:50.368937 :: got:iteration 3
2024-08-19 16:18:50.368966 :: got:iteration 4
2024-08-19 16:18:50.368994 :: got:iteration 5

# unbuffered
python bgen.py stdbuf
2024-08-19 16:18:58.908986 :: running: ['/usr/bin/stdbuf', '-o0', '/home/tester/someprocess']
2024-08-19 16:18:58.910353 :: got:iteration 0
2024-08-19 16:18:59.910556 :: got:iteration 1
2024-08-19 16:19:00.910656 :: got:iteration 2
2024-08-19 16:19:01.910810 :: got:iteration 3
2024-08-19 16:19:02.910920 :: got:iteration 4
2024-08-19 16:19:03.911044 :: got:iteration 5

hope this helps ...

answered Aug 19, 2024 at 15:30

ticktalk

1,2071 gold badge4 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

bgen Over a year ago

I should have specified but i'm looking to get this working on windows.

ticktalk Over a year ago

@bgen, np, i am not a windows user, but believe the WSL is available for Windows 10 - if still unsuitable just ignore my post.

Collectives™ on Stack Overflow

Reading stdout of non-python subprocess continuously

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related