Realtime output using python subprocess of script with read prompts

Question

I have a bash script which interacts with the user using read prompts, for example:

#!/bin/bash
echo start
read -p "Press something to continue " -n 1 -r
echo got $REPLY.
read -p " Press again " -n 1 -r

echo " done"

I want to run the script as a subprocess using python, while both capturing the output and displaying it in real time to the user. Browsing various threads here, I have come up with this solution that does work for me, however very slowly if the script produces large outputs (significantly longer than running the script manually), probably due to the continuous usage of flush for every character:

import subprocess

with subprocess.Popen(['./a.sh'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) as proc:
  output_text = ''
  while c := proc.stdout.read(1):
    print(c, end='', flush=True)
    output_text += c

The "read(1)" is needed since read is blocking and trying to read larger chunks means that the prompt message won't be fully displayed before the user is expected to enter input. The flush is also necessary since without it the input prompt message is only printed after the user is required to enter input.

I'm looking for a solution that will allow reading large chunks of data (hence reducing the number of calls for print), while still making sure that the full prompt appears to the user before waiting for their input.

Also, for the purpose of this question, the bash script is given as input and cannot be changed, nor do we have any information about its structure. I'm trying to write code that will generically execute such scripts while both presenting and capturing their output.

Similarly to using larger chunks, readline wouldn't work here as it expects EOL, but EOL is only received after the user enters input for the prompt.

I've tried looking at pexpect, but all the examples I've seen assume they know which prompt to look for, while in this case I assume no knowledge of the script I'm running.

In various similar questions I've seen, the solution either uses a single char read (e.g., by calling read(1) or setting buffer size to 1), or uses readline - non of which seem suitable for my current use case. examples: How to get realtime Python subprocess output? Getting realtime output using subprocess Getting realtime and full output using subprocess

Armali · Accepted Answer · 2023-10-19 07:17:08Z

1

Instead of proc.stdout.read(1) you can from os import read and read(proc.stdout.fileno(), 4096) - this will return also if less than the given length is readable.

answered Oct 19, 2023 at 7:17

Armali

19.6k15 gold badges64 silver badges185 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user1656308 Over a year ago

Thanks a lot! worked like a charm. One small additional change I had to do is use c.decode() since os.read returns bytes whereas proc.stdout.read(1) returns str if Popen is created with text=True, but other than that it's exactly what I needed.

Collectives™ on Stack Overflow

Realtime output using python subprocess of script with read prompts

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related