0

I have the following script:

#!/usr/bin/perl -w
use strict;
$| = 1;
foreach (1..5) {
    print $_ . "\r";
    sleep 1;
}
print "\n";

This behaves as expected: the numbers 1,2,3,4,5 overwrite each other on the console.

$ ./loop.pl | hexdump -C
00000000  31 0d 32 0d 33 0d 34 0d  35 0d 0a                 |1.2.3.4.5..|

However, a different script (intended to show hide the large output of a long-running program like this: long_running_program | tee output | ./progr)

#!/usr/bin/perl -w
use strict;
$| = 1;
while (<>) {
    chomp;
    print $_ . "\r";
}
print "\n";

produces different behavior when input is redirected:

 perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl

No output is seen for five seconds, then a '5' can be seen. Yet hexdump shows the same output (after five seconds)

$ perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl | hexdump.exe -C
00000000  31 0d 32 0d 33 0d 34 0d  35 0d 0a                 |1.2.3.4.5..|

This is not Perl-specific. The following C code

for (int i = 0; i < 6; ++i) {
    printf("%d\r", i);
    fflush(stdout);
    sleep(1);
}
puts("\n");

shows digits overwriting each other, but

#define SIZE (256 * 1024)
char buffer[SIZE];
int line = 0;
while (fgets(buffer, SIZE, stdin)) {
    printf("%d\r", ++line);
    fflush(stdout);
}
puts("\n");

when at the end of a pipe, only shows output after the input is exhausted.

Not even

setvbuf(stdout, NULL, _IONBF, 0);

appears to help.

I tried all these through an SSH connection to a remote Linux (RHEL6) system, and locally under Cygwin.

(Edited with corrections from @Fredrik and @usr)

6
  • printf(%d\r", i); You are missing a quote ” Commented Nov 8, 2019 at 22:48
  • You should make stdout unbuffered (not stdin): setvbuf(stdout, NULL, _IONBF, 0);. Commented Nov 8, 2019 at 22:53
  • setvbuf(stdout, NULL, _IONBF, 0) doesn't help either :( Commented Nov 8, 2019 at 22:58
  • @Bulletmagnet This stdbuf -i0 -o0 -e0 long_running_program | tee output | ./progr? Commented Nov 8, 2019 at 23:06
  • The question does not have a minimal reproducible example for the C case. Pipes are not line-buffered. perl -wle does not flush Commented Nov 9, 2019 at 2:37

2 Answers 2

2

You're looking at the wrong program. You turned off output buffering in the second program of the pipeline, but not the first.


STDOUT is line-buffered if connected to a terminal, block-buffered otherwised.

Line-buffered: Flushed when a Line Feed is output.

Block-buffered: Flushed when the buffer is filled.

Since the STDOUT of the first program of the pipeline (the input generator) is connected to a pipe, its output is block-buffered. And since the buffer is large enough to hold the entirety of the program's output, your input generator doesn't actually output anything until it exits.

Change

perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl

to

perl -wle '$| = 1; foreach (1..5) { print $_; sleep 1 }' | ./progr.pl

But what if you have no control over the program? You can sometimes coax other program to turn off buffering by using unbuffer.

unbuffer perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks for a well-written explanation, and for the suggestion to use unbuffer, which worked.
-1

In most of those examples, you're piping output that only has carriage returns (until the program ends and prints a newline (Or two for the C version)) to programs that are reading a newline-terminated line at a time. Of course you're not going to see output from the second program until the first program ends -- it's waiting on a newline or end of file to return the first line.


In the perl -wle 'foreach (1..5) { print $_; sleep 1 }' | ./progr.pl case, yes, there are newlines thanks to the -l option, but the output is buffered because it's to a pipe, and progr.pl doesn't see any input until the first part finishes. Add a $|=1; at the beginning before the loop and you'll get different results.

2 Comments

The Perl one-liner's output has no carriage returns. It has newlines. Only the second program outputs carriage returns.
@Bulletmagnet The one-liner also buffers its output because stdout is a pipe. The progr.pl script doesn't see any input until the one-liner's finished. You have to flush output; see edit.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.