python calling perl, file isn't created by Popen until read()

Question

Edit: Oops! I accidentally posted code that didn't match my question. I started writing this post before I was done experimenting, so I posted code from an intermediate point in my testing process. I've amended this question to reflect what I really meant to post by changing the code slightly.

The following Python code works as expected with one problem: unless I If verbose == False, the file produced by the Perl script is not created. Why do I have to call output.stdout.read() for the underlying Perl script to successfully create a file?

cmdStringList = ["perl","script.pl","arg1",...]

output = subprocess.Popen(cmdStringList,stdout=subprocess.PIPE)

if verbose:
    print output.stdout.read()

I didn't even realize anything was wrong until I tried to execute my Python script with verbose=False in a production environment. I did some google-fu to try to understand the behavior of Popen and subprocess, but I haven't come up with a reason for this behavior. Any help would be greatly appreciated.

Your question is inconsistent. You said not calling output.stdout.read() causes it to fail, and you also said it started to fail when verbose was set to false. Which one is it? Did you post the wrong code? — ikegami
– ikegami, Commented Feb 7, 2013 at 0:26
You're right - I messed up my question. I changed it to have the code that I initially meant to post. — Anthony
– Anthony, Commented Feb 7, 2013 at 3:53

ikegami · Accepted Answer · 2013-02-07 08:45:53Z

1

You actually have the following code, right?

if verbose:
    outputRead = output.stdout.read()
    print outputRead

It's quite likely that the problem is that the child is blocked trying to write to STDOUT. Until you make some space in the pipe by reading from it, the child won't be able to finish writing to STDOUT and move on to where it creates the file to which you refer.

If you want to prevent the child from blocking without reading, redirect its stdout to nul (Windows) or /dev/null (elsewhere). This should work everywhere but Windows:

cmdStringList = ["sh","-c","\"$0\"" \"$@\" >/dev/null","perl","script.pl","arg1",...]

(Pardon any syntax errors. I don't know Python at all.)

edited Feb 7, 2013 at 8:45

answered Feb 7, 2013 at 0:33

ikegami

391k17 gold badges291 silver badges555 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Anthony Over a year ago

Yes, you have it right. My initial code was accidentally wrong because of a timing error on my part. I didn't transfer it across a variable in the code I meant to post, though. I directly printed output.stdout.read(), which is now shown in my question.

Anthony Over a year ago

This makes sense I think. I'm not sure how to confirm that it's the problem, though.

ikegami Over a year ago

Does the Perl program print stuff out before creating the file? If so, there you go.

Community · Accepted Answer · 2017-05-23 12:18:58Z

1

You should keep reading a stream, otherwise the buffer may become full, then your perl process may get blocked. It has nothing to do with a specific language but the underyling operating system.

I encountered a similar problem when I use Java.

edited May 23, 2017 at 12:18

CommunityBot

11 silver badge

answered Feb 7, 2013 at 1:28

StarPinkER

14.3k8 gold badges57 silver badges81 bronze badges

2 Comments

Anthony Over a year ago

How do I continuously read the pipe if I don't want to block program execution? Should I use the mutliprocessing module?

ikegami Over a year ago

That would be a waste. Just redirect the output to nul or /dev/null, like I mentioned.

Collectives™ on Stack Overflow

python calling perl, file isn't created by Popen until read()

2 Answers 2

3 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related