Unexpected broken pipe error in shell pipeline inside Jupyter Notebook terminal

Question

I encountered ls: write error: Broken pipe when I run ls -l | head in a directory containing too many files. I did not encounter that problem when running ls | head. I just wonder why -l option would cause that problem. Also I would like to know how to resolve that problem but still with long information of files printed.

To reproduce my issue, you can first open a within terminal provided by Jupyter Notebook,

and then run the following codes.

cd
mkdir test
cd test
for i in {1..200}; do touch $i; done
ls -l | head

Output is as follows.

total 0
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 1
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 10
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 100
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 101
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 102
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 103
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 104
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 105
-rw-rw-r-- 1 s1855106 s1855106 0 Apr 14 10:14 106
ls: write error: Broken pipe

I cannot reproduce this, but have a look at How can I fix a broken pipe error. Possibly your shell (and hence any command started by it) is ignoring SIGPIPE (as an illustration, (trap '' PIPE; ls -l | head) will reliably reproduce your issue). You can see the signal disposition of your bash shell with the builtin trap -p — Hans Lub
– Hans Lub, Commented Apr 14, 2020 at 9:50
I can't reproduce this error - I tried with 20,000 files, but ls -l | head works as expected - could be something to do with your shell (bash v5?) or your ls command (gnu ls?) ? — jared_mamrot
– jared_mamrot, Commented Apr 14, 2020 at 9:53
Thank you all for your very helpful comments! I have now found that I will encounter this problem if I use the terminal provided by Jupyter Notebook. All the things are fine if I use other terminal like Git Bash. I have edited my problem accordingly and I appreciate your further help. I am using Jupyter Notebook Terminal mainly because I want to do all the things on web pages. — Fei Yao
– Fei Yao, Commented Apr 14, 2020 at 10:52

Hans Lub · Accepted Answer · 2020-04-15 12:08:08Z

0

Python (at least older versions of python2) doesn't reset SIGPIPE handling for subprocesses

This means that any commands started by it (e.g. commands you start in a Jupyter notebook) will ignore SIGPIPE, exactly as if you had issued trap '' PIPE. This, in turn, means that commands in a pipeline don't catch the SIGPIPE that is sent to them when they try to write to their stdout while the following command (tail, in your case) in the pipeline has finished, and thus see a write error instead. See, for example, this bug report.

(trap '' PIPE; ls -l | head) will reliably reproduce your problem in any (non-Jupyter) shell.

Apparently, python3 has fixed this, so you could try changing the python version used by your notebook. Alternatively, you could try "un-ignoring" SIGPIPE by issuing trap - PIPE

edited Apr 15, 2020 at 12:08

answered Apr 14, 2020 at 10:59

Hans Lub

5,7281 gold badge27 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Unexpected broken pipe error in shell pipeline inside Jupyter Notebook terminal

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related