How to capture stderr from Python mulitprocessing process

Question

How do you capture the stdout/stderr of a specific target of a process launched from Python's mulitprocessing.Process library, ideally without requiring special modifications to the target function?

e.g.

p = Process(target=myfunc)
p.daemon=True
p.start()
p.join()
print p.exitcode, p.stderr.getvalue()

I know how to override sys.stdout/sys.stderr with a custom file object and use multiprocessing.Queue to redirect IO between processes, but this requires modifying the target function to write its output to the Queue.

Is something like this available with Process? I'm assuming it is on some level, since I'm able to see the process's output in the terminal when running, even if I can't record it from the parent process.

If you want an error message have a look at stackoverflow.com/questions/13535680/… although that may not fit your use case. — User
– User, Commented Feb 8, 2014 at 9:42

User · Accepted Answer · 2014-02-08 19:19:00Z

This is no answer but a long comment.

I would create a wrapper function that replaces sys.stderr and writes it to a file/pipe and then after/while the process ran the stderr is read into the parent. Reason:

This seams difficult if you do not alter the function

Multiprocessing does not use subprocess. You subclass Process and set Process.Popen to replace multiprocessing.forking.Popen. Replacing multiprocessing.forking.Popen must implement different code for the platforms 'win32' and 'not win32' leading to two different subclasses OR use subprocess.

Modifying multiprocessing.forking.Popen to use subprocess should be your best try from my point of view.

In Process.start()

from .forking import Popen
self._popen = Popen(self) # also self.Popen

In multiprocessing.forking

class Popen(object):
    def __init__(self, ...):
        # ...
        # for windows
        hp, ht, pid, tid = _subprocess.CreateProcess(
            _python_exe, cmd, None, None, 1, 0, None, None, None
            )

Now having a look at subprocess:

(p2cread, p2cwrite,
     c2pread, c2pwrite,
     errread, errwrite) = self._get_handles(stdin, stdout, stderr)
self._execute_child(args, executable, preexec_fn, close_fds,
                            cwd, env, universal_newlines,
                            startupinfo, creationflags, shell,
                            p2cread, p2cwrite,
                            c2pread, c2pwrite,
                            errread, errwrite)
self.stdout = os.fdopen(c2pread, 'rU', bufsize)

Collectives™ on Stack Overflow

How to capture stderr from Python mulitprocessing process

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related