Converting file descriptor (pipe) to file object in python

Question

I'd like to use anonymous pipes to communicate pickled objects between two processes in linux. I don't want to use the subprocess module and instead manually create the necessary pipes and then call pickle.dump and pickle.load. The last two functions, however, require a file object. So, while testing things around, I was faced with the following problem. The following code works fine:

import os
r, w = os.pipe()
os.write(w, b"test")
os.read(r, 1024)

but this code stalls at the read operation

import os
r, w = os.pipe()
fr = os.fdopen(r, "rb")
fw = os.fdopen(w, "wb")
fw.write(b"test")
fr.read()

Additionally, select.select([r],[],[],0) shows the pipe is empty. My question: what is going on during file object creation and why doesn't it work for pipes? Is there a way to get a file object this way at all?

please always use the generic [python] tag when asking Python questions — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Mar 13, 2019 at 23:42

DYZ · Accepted Answer · 2019-03-13 22:22:40Z

3

You have a combination of two problems. First, a file object created by os.fdopen is buffered by default. Either make it unbuffered:

fr = os.fdopen(r, "rb", buffering=0)
fw = os.fdopen(w, "wb", buffering=0)

Or flush the written data:

fw.write(b"test")
fw.flush()

Second, function fr.read() when called without parameters reads to the end of file. In the case of a pipe, it means, until the pipe is closed. You should pass the number of bytes you want to read:

fr.read(4)
> b'test'

If you are not sure how much data will be written, read it piecewise, N bytes per call, and then recombine.

answered Mar 13, 2019 at 22:22

DYZ

57.3k10 gold badges73 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ivan Over a year ago

Thank you, this makes a lot of sense (and is in the documentation for the built-in open which I should probably pay closer attention to :-))

Collectives™ on Stack Overflow

Converting file descriptor (pipe) to file object in python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related