systemd unit file never finishes when using a shell command with pipes

Question

I'm trying to execute a shell script with systemd.

If I run my script from the bash everything works fine. But if I run the same script via systemd it never finishes. The command where it seems to hang is:

random="$(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w128 | head -n1)"

If I'm replacing this line with random="1234" it also runs with systemd. I guess the 'hanging' command is the tr - its process never finishes.

And this is the systemd unit file I'm using:

[Unit]
Description=my script

[Service]
Type=forking
Restart=on-failure
ExecStart=/mypath/script.sh start
ExecStop=/bin/kill $MAINPID
ExecStopPost=/bin/rm -f /mypath/RUNNING_PID

[Install]
WantedBy=multi-user.target

Does your service fork? If not, then Type=fork is wrong service type. See man systemd.service for options. — Mark Stosberg
– Mark Stosberg, Commented Jun 5, 2017 at 15:22
Restart=on-failure will restart it if systemd sees that the script exited with non zero status. Is random="$(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w128 | head -n1)" the last command in the script? — alvits
– alvits, Commented Jun 5, 2017 at 21:10
@alvits Yes, I want to restart the service if it crashes. And: the tr is not the last command in the script. — Kris
– Kris, Commented Jun 6, 2017 at 20:12

U. Windl · Accepted Answer · 2022-09-20 08:18:34Z

13

Edit: Made the explanation clearer and added new information.

Short answer: systemd is filtering SIGPIPEs, so set IgnoreSIGPIPE=false under [Service] in the .service file. From the systemd.exec manual:

   IgnoreSIGPIPE=
       Takes a boolean argument. If true, causes
       SIGPIPE to be ignored in the executed process.
       Defaults to true because SIGPIPE generally is
       useful only in shell pipelines.

Long explanation:

random="$(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom | fold -w128 | head -n1)"

When the head command exits after the first newline received from fold, its open file descriptors are closed. Thus when the fold command later tries to write to the pipe, it will receive a SIGPIPE signal (as described in pipe(7)). The default action for this signal is a termination of the process. That mechanism repeats towards the start of the pipeline, so the fold command would be terminated next, and eventually the tr command.

However, when the pipeline is run under systemd, systemd sets the default action for SIGPIPE to SIG_IGN, which makes the processes in the pipeline ignore the signal.

When fold (and the other components of the pipeline) inherits that signal handler, it won't be notified (killed) via SIGPIPE when writing to the now closed output pipe (Instead each write to the pipe will return an EPIPE error).

As fold command does not check the return value of those write calls (at least not in coreutils-8.26), this makes fold continue reading from the pipe (stdin) and to write to the closed pipe (stdout), even though an error is reported each time. So fold keeps its input pipe from tr open, and tr will happily write to the output pipe feeding fold forever.

edited Sep 20, 2022 at 8:18

U. Windl

4,74839 silver badges65 bronze badges

answered Jun 5, 2017 at 20:02

Tom Bjerck

2563 silver badges4 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

alvits Over a year ago

This Restart=on-failure affects the behavior. You already figured out the exit status.

Kris Over a year ago

Thanks Tom! I looked up IgnoreSIGPIPE and it looks like it defaults to true: freedesktop.org/software/systemd/man/…. But what you wrote before - before you edited the answer - actually helped me :). If I use $(LC_ALL=C tr -cd '[:alnum:]' < /dev/urandom 2>/dev/null | dd bs=128 count=1 2>/dev/null) it works. I exchanged fold -w128 | head -n1 with dd bs=128 count=1 like you suggested before. And I had to add 2>/dev/null to get rid of the tr: write error: Broken pipe that was in journalctl. So I'd check your answer as correct as it was before :)

alvits Over a year ago

Your original answer is a workaround. But this revised answer is definitely the answer. Thanks for quoting the man page, it helps everyone in the community.

Collectives™ on Stack Overflow

systemd unit file never finishes when using a shell command with pipes

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related