Reading from pipe Python

Question

I have a process that gathers information from the web and stores into files. I have another process that has the purpose to use this data to do analysis. That is, P1 writes the data to files and P2 used these information to do analysis.

Now my problem is with P2. How can I make P2 get input from the data files as a stream? E.g when the bytes are received from P1 to be also input to P2.

I am thinking of using IPC or something similar to that. I thinked of using PIPES but the problem is that the P1 may block until P2 reads the data. The P1 should never stop receiving data. The P2 can be extremely late. One idea is to use a different thread to send the data to the pipe until there is a time out.

Is this approach correct?

Threads within a process are subject to the same limitations as the processes themselves. If P1 and P2 are unrelated processes then indeed P1 could block if P2 stops reading, or takes a much longer time to consume the data than it takes for P1 to produce it. In the general case, you cannot have an unlimited amount of queue. Is there any way for you to add more parallel P2 processes? — tripleee
– tripleee, Commented Feb 4, 2014 at 13:44

GabiMe · Accepted Answer · 2014-02-04 13:40:20Z

1

Seems that indeed named pipes are your tool for this.

Very easy to consume as regular files from python or using python's os module

answered Feb 4, 2014 at 13:40

GabiMe

18.5k29 gold badges80 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Reading from pipe Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related