arrange disrupted output in cmd when Read streaming output from async subprocess

Question

This code runs in Pycharm without any problems but when I want to run in Command Prompt outputs arrange disrupted

I have two files called test.py and test1.py whose codes are as follows. test1.py is executed by test.py and its output is displayed at test.py console.

test1.py

import logging
import time

logging.error(1)
time.sleep(1)
print(2)
time.sleep(1)
print(3)
time.sleep(1)
logging.error(4)
time.sleep(1)
logging.error(5)
time.sleep(1)
print(6)

test.py

import asyncio


async def _read_stream(stream, cb):
    while True:
        line = await stream.readline()
        if line:
            cb(line)
        else:
            break


async def _stream_subprocess(cmd, stdout_cb, stderr_cb):
    process = await asyncio.create_subprocess_exec(*cmd,
                                                   stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
    stdout_task = asyncio.create_task(_read_stream(process.stdout, stdout_cb))
    stderr_task = asyncio.create_task(_read_stream(process.stderr, stderr_cb))

    await asyncio.wait([stdout_task, stderr_task])
    return await process.wait()


def execute(cmd, stdout_cb, stderr_cb):
    asyncio.run(_stream_subprocess(cmd, stdout_cb, stderr_cb))


if __name__ == '__main__':
    print(execute(["pipenv", "run", "python", 'test1.py'],
                  lambda x: print("STDOUT: %s" % x),
                  lambda x: print("STDERR: %s" % x),))

When I run the test.py in PyCharm, the program runs without problems and its output is as follows:

STDERR: b'ERROR:root:1\r\n'
STDOUT: b'2\r\n'
STDOUT: b'3\r\n'
STDERR: b'ERROR:root:4\r\n'
STDERR: b'ERROR:root:5\r\n'
STDOUT: b'6\r\n'

But when I run the test.py in Command Prompt, the output arrangement is disturbed and STDOUT is not properly executed. its output is as follows:

STDERR: b'ERROR:root:1\r\n'
STDERR: b'ERROR:root:4\r\n'
STDERR: b'ERROR:root:5\r\n'
STDOUT: b'2\r\n'
STDOUT: b'3\r\n'
STDOUT: b'6\r\n'

Can anyone help me to solve my problem?

jsbueno · Accepted Answer · 2023-04-06 21:29:04Z

The problem seems to be that in some your environments reading from the stdin already has the data available, so line = await stream.readline() won't really make the asyncio loop through other tasks: if the data is there, it simply will return synchronously.

So your tasks for stdin and stderr will run one after the other.

I don't think fixing this will ensure you a determistic order, though - you really should just pipe your stdin and stderr messages to a common queue instead of relying on asyncio tasks to yield the messages in the order they show up.

Anyway, adding an extra asyncio.sleep call in your _read_stream() function should give you the same behavior in both environments:

async def _read_stream(stream, cb):
    while True:
        line = await stream.readline()
        await asyncio.sleep(0)  # Ensures the asyncio loop steps through all ready tasks
        if line:
            cb(line)
        else:
            break

Yes - I also thought any await would always give a chance to other tasks to run - last week, when answering a similar question here I found out it not to be so - check my lengthier answer at: Why sending a message to a web socket does not yield control to the event loop?

Collectives™ on Stack Overflow

arrange disrupted output in cmd when Read streaming output from async subprocess

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related