how do I run concurrent background process from shell script?

Question

I tried the following:

#!/bin/bash

while read device; do
        name=$(echo "$device" | awk '{ print $1 }')
        ip=$(echo "$device" | awk '{ print $2 }')
        while read creds; do
                community=$(echo "$creds" | awk '{ print $1 }')
                version=$(echo "$creds" | awk '{ print $2 }')
                mkdir -p walks/$name;
                `echo -e "snmpwalk -v$version -c \x27$community\x27 $ip system > walks/$name/$community-$version.txt
        done < <(##MySQL query that returns tuples in form: (snmp_ro,(1,2c,3))##")
done < <(cat devices.txt)
exit 0

This is meant to go through and find the snmp string and version of each device.

devices.txt is a list of devices in form: hostname ip

It doesn't create the file: walks/$name/$community-$version.txt, and it only seems to run through the walks 1 at a time, something I don't want.

And the backticks around the echo command is just broken; that's not how you background a process in bash. — Charles Duffy
– Charles Duffy, Commented Sep 21, 2015 at 19:48
@chepner, check spelling; I assume you mean shellcheck.net :) — Charles Duffy
– Charles Duffy, Commented Sep 21, 2015 at 19:48
Also, read creds is silly when you're just going to split the contents with awk. Why not read community version _, and get rid of the awk? — Charles Duffy
– Charles Duffy, Commented Sep 21, 2015 at 19:49

Charles Duffy · Accepted Answer · 2015-09-21 23:03:23Z

2

Use & to put the contents you want backgrounded in, well, the background.

pids=( )
while read -r -u 3 name ip _; do
  while read -r -u 4 community version _; do
    mkdir -p "walks/$name"
    snmpwalk -v"$version" -c "$community" "$ip" system \
      </dev/null >"walks/$name/$community-$version.txt" & pids+=( "$!" )
  done 4< <(: get data for "$name" and "$ip")
done 3<devices.txt
wait "${pids[@]}"

Other items of note:

read can already split fields into their own variables; using awk for this is silly.
The _ in read -r foo bar _ ensures that if more than two columns exist in the input file, the third column and onward are discarded (actually, put into a variable named _, but this is considered discard by convention) rather than appended to bar.
Make a habit of quoting expansions unless you have a specific and compelling reason to do otherwise; otherwise, you get string-splitting and glob expansion of string contents.
This example puts each input stream on its own file descriptor, and redirects each read to its own FD. This prevents any other content within your loop from consuming stdin.

edited Sep 21, 2015 at 23:03

answered Sep 21, 2015 at 19:53

Charles Duffy

299k43 gold badges441 silver badges497 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

ojblass Over a year ago

I would add nohup to it as well for completeness. The background task will terminate with the shell otherwise.

Charles Duffy Over a year ago

@ojblass, I'd only go there if the OP explicitly wants that behavior. Often, it's better to track a list of PIDs and wait for them, so the shell script doesn't exit until its children are done. (This also allows better error reporting).

Charles Duffy Over a year ago

@ojblass, also, even if that's what you want, nohup isn't the best way to get there. See the shell builtin disown, particularly including its option -h. (Note, too, that huponexit is an option, not constantly enabled, on-by-default behavior).

Charles Duffy Over a year ago

@ojblass, ...I've extended the example here to show the explicit tracking approach, albeit not the fancier version (tracking a relationship between the PIDs and their individual output files to allow the output files to be deleted if the command generating their content failed).

Charles Duffy Over a year ago

@ojblass, ...also, it's not accurate that a script exiting will kill all background processes it spawned. A terminal exiting will do that, for processes which have stdin, stdout or stderr connected to it, but that can be resolved simply by redirecting the relevant FDs -- which is most of what nohup does (the remainder being suppressing HUP-on-exit behavior, but that's off by default in bash anyhow).

Collectives™ on Stack Overflow

how do I run concurrent background process from shell script?

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related