191

I have the following shell script. The purpose is to loop thru each line of the target file (whose path is the input parameter to the script) and do work against each line. Now, it seems only work with the very first line in the target file and stops after that line got processed. Is there anything wrong with my script? 

#!/bin/bash
# SCRIPT: do.sh
# PURPOSE: loop thru the targets 

FILENAME=$1
count=0

echo "proceed with $FILENAME"

while read LINE; do
   let count++
   echo "$count $LINE"
   sh ./do_work.sh $LINE
done < $FILENAME

echo "\ntotal $count targets"

In do_work.sh, I run a couple of ssh commands.

Improve this question
6
  • 1
    Your script is fine, but there might be something wrong with the do_work.sh Commented Dec 10, 2012 at 11:43
  • 2
    Yes, it could eat up all input, or it could be invoked as source and simply exit or exec. But this code doesn't look genuine, the OP would notice that echo requires -e to display line feed properly... Commented Dec 10, 2012 at 11:46
  • 3
    Does do_work.sh run ssh by any chance? Commented Dec 10, 2012 at 11:48
  • 1
    yes, do_work.sh runs a couple ssh commands. anything special about that? Commented Dec 10, 2012 at 11:50
  • 1
    Better you show the do_work.sh source and also run do.sh with set -x to debug. Commented Dec 10, 2012 at 11:56

6 Answers 6

Reset to default
299

The problem is that do_work.sh runs ssh commands and by default ssh reads from stdin which is your input file. As a result, you only see the first line processed, because the command consumes the rest of the file and your while loop terminates.

This happens not just for ssh, but for any command that reads stdin, including mplayer, ffmpeg, HandBrakeCLI, httpie, brew install, and more.

To prevent this, pass the -n option to your ssh command to make it read from /dev/null instead of stdin. Other commands have similar flags, or you can universally use < /dev/null.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Very useful, helped me to run this zsh oneliner: cat hosts | while read host ; do ssh $host do_something ; done
@rat You still want to avoid the useless cat. You'd think a rodent in particular would be wary of this.
while read host ; do $host do_something ; done < /etc/hosts would avoid it. That's quite a life saver, thanks!
httpie is another command that reads STDIN by default, and will suffer from the same behavior when called inside a bash or fish loop. Use http --ignore-stdin or set standard input to /dev/null as above.
What will be the solution for gcloud compute ssh case?
79

A very simple and robust workaround is to change the file descriptor from which the read command receives input.

This is accomplished by two modifications: the -u argument to read, and the redirection operator for < $FILENAME.

In BASH, the default file descriptor values (i.e. values for -u in read) are:

  • 0 = stdin
  • 1 = stdout
  • 2 = stderr

So just choose some other unused file descriptor, like 9 just for fun.

Thus, the following would be the workaround:

while read -u 9 LINE; do
   let count++
   echo "$count $LINE"
   sh ./do_work.sh $LINE
done 9< $FILENAME

Notice the two modifications:

  1. read becomes read -u 9
  2. < $FILENAME becomes 9< $FILENAME

As a best practice, I do this for all while loops I write in BASH. If you have nested loops using read, use a different file descriptor for each one (9,8,7,...).

Improve this answer

6 Comments

Thanks! this is actually the best solution here IMO
Thank you very much! This looks also like the best solution for me right know because with this one you don't need to find the problematic command if you have larger scripts.
This is a really nice trick. It can be quite a surprise that arbitrary commands end up reading stdin. In my case it was sloccount. This is general solution that doesn't require reading command specific man pages (if any).
This is IMHO a better solution than the accepted one. There are other commands besides ssh that read stdin, messing up the while loop. This solution, to use a different file descriptor, allowed me to keep existing called infrastructure in the loop the same. This allowed me to avoid having to test interactions with sshpass, or other programs in the loop that might affect stdin. Thanks!!
The -u option to read is a bashism; to make this portable to shells other than bash, just use while read LINE <&9; do.
|
38

More generally, a workaround which isn't specific to ssh is to redirect standard input for any command which might otherwise consume the while loop's input.

while read -r line; do
   ((count++))
   echo "$count $line"
   sh ./do_work.sh "$line" </dev/null
done < "$filename"

The addition of </dev/null is the crucial point here, though the corrected quoting is also somewhat important for robustness; see also When to wrap quotes around a shell variable?. You will want to use read -r unless you specifically require the slightly odd legacy behavior you get for backslashes in the input without -r. Finally, avoid upper case for your private variables.

Another workaround of sorts which is somewhat specific to ssh is to make sure any ssh command has its standard input tied up, e.g. by changing

ssh otherhost some commands here

to instead read the commands from a here document, which conveniently (for this particular scenario) ties up the standard input of ssh for the commands:

ssh otherhost <<'____HERE'
    some commands here
____HERE
Improve this answer

Comments

5

ssh -n option prevents checking the exit status of ssh when using HEREdoc while piping output to another program. So use of /dev/null as stdin is preferred.

#!/bin/bash
while read ONELINE ; do
   ssh ubuntu@host_xyz </dev/null <<EOF 2>&1 | filter_pgm 
   echo "Hi, $ONELINE. You come here often?"
   process_response_pgm 
EOF
   if [ ${PIPESTATUS[0]} -ne 0 ] ; then
      echo "aborting loop"
      exit ${PIPESTATUS[0]}
   fi
done << input_list.txt
Improve this answer

1 Comment

This doesn't make sense. The <<EOF overrides the </dev/null redirection. The << redirection after the done is wrong.
4

This was happening to me because I had set -e and a grep in a loop was returning with no output (which gives a non-zero error code).

Improve this answer

2 Comments

You beauty, just saved my afternoon
Perhaps see stackoverflow.com/questions/19622198/… for more details about set -e
2

I had a similar problem but I was not reading from a file but from a bash variable containing some csv data, so the file descriptor thing did not work for me.
My solution was to feed some empty input into the program that was stealing data from stdin via
echo -n | input_stealing_program

Example:

csv_data="Germany,Berlin
France,Paris
Enland,London"

# Set IFS to comma for parsing CSV
IFS=','

while read -r country capital; do
    echo -n | input_stealing_programm $country $capital
done <<< "$csv_data"

An alternative would be input_stealing_program < /dev/null

Improve this answer

3 Comments

That's just a painful paraphrase of </dev/null
How is it painful?
I certainly writhe when I see beginners write needlessly complex shell script; don't you? The useless extra process doesn't cost much in this individual case, but put it in a tight loop and it starts to matter. Additionally, echo -n has portability problems - if you really are using Bash it doesn't matter, of course; but many beginners are confused about the difference between sh and bash.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.