3

In a bash script I iterate through folders looking for some files, and if I find them I call a function with the directory that holds those files. See below

pairedread $1 &
pairedread $2 &
pairedread $3 & 
wait
echo "Done ..."

echo 
echo "======================"
echo "Testing again"
echo "======================"
echo

find . -type d -print | while read DIR; do
    echo "reading..."
    test -r "$DIR"/*_1.gz -a -r "$DIR"/*_2.gz || continue
    ( pairedread $DIR &  )
    done

wait
echo "Done..."

pairedread is the function that takes the folder and calls a python script on the files that are in the given directory. In the first case, ie when I explicitly give the folders that contain the files of interest, the script runs with instances of pairedread executing and finally terminating followed with the helpful message "Done..." after all subprocceses complete.

In the second case, the same three directories are picked and three instances of pairedread are created. However the script doesn't wait at all, it prints "Done..." immediately and returns while the subprocesses are running in the background.

Am I missing something? Why can't I wait for the subprocesses to finish before going on with the script?

Improve this question
2
  • Is there are reason for encapsulating the backgrounding with subshell in the second case? Commented Mar 29, 2016 at 14:51
  • I'm going to guess that the wait in the outer doesn't know about the inner children since they're an extra layer removed because of the pipe. Try redoing the while to read from stdin the output of find and see if that helps? Commented Mar 29, 2016 at 14:51

2 Answers 2

Reset to default
3

Because you're launching those processes in a subshell (because of the pipe and because of the extra (...) options both) the wait doesn't know about any children to wait for. You can rewrite that loop so it doesn't require a subshell like:

while read DIR; do
    echo "reading..."
    test -r "$DIR"/*_1.gz -a -r "$DIR"/*_2.gz || continue
    pairedread $DIR &
done < <(find . -type d -print)

Wait only knows about children of the current process. When you use | it creates a subshell for the parts to connect the stdin/stdout of the sides together. Any process launched in a subshell is not a child of the "top" process, so wait doesn't know about them.

So in this case you were thwarted both by explicicly launching your pairedread in a subshell with () syntax, and also by having it happen within the block of a while loop in a pipeline.

Rerwriting that block to avoid the pipeline and removing the explicit subshells lets the outer wait know about the child processes and so do what you expected

Improve this answer
1

As @Eric pointed out, you need to background outside the pipe and not subshell the command, if you want to wait on the process.

Double fork: When you do a minimum of two forks, the process will be orphaned and the init process becomes the parent of the process.

Your code: You did three forks(pipe, subshell and background), because of this, init process becomes parent of your newly started process and you cannot wait on it, as wait can only wait on its own child process.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.