Executing a for loop with gnu parallel

Question

I'd like to use gnu parallel to exec a function inside a bash script in a for loop and I'm not able to figure out how from the info I got online . For example:

#!/bin/bash

get_racks(){
query that outputs a list of racks 
}

get_hosts() {
query that outputs a list of hosts by rack(passed as param to the func)
}

get_gw() {
query that outputs a list of gateways
}

check_ping() {
HOSTS=(`get_hosts`)
COUNTER=0
while [ $COUNTER -lt $SIZE ] ; do
  ssh $HOSTS "ping -c 5 ${GATEWAYS[$COUNTER]} " ;
  COUNTER=$((COUNTER+1)) ;
done
}

RACKS=(`get_racks`)
HOSTS=(`get_hosts`)
GWS=(`get_gw`)
SIZE=${#GWS[@]}
COUNTER=0

for rack in ${RACKS[@]}; do
  parallel check_ping | tee output.txt 
done

I'd like to execute the for loops and what happens inside in parallel while controlling the number of concurrent jobs and output everything into a single file . The final file should have all the data the function returns and also print to stdout . Also , the check_ping has a part of sshing to a remote machine and pinging a GW, this part happens actually in parallel with a custom ssh command we have that ssh to all the hosts in the array in parallel and pings a GW .

Thanks much

Mark Setchell · Accepted Answer · 2017-11-25 08:55:15Z

2

I think you want this:

RUN_COMMANDS() {
...
}

export -f RUN_COMMANDS

parallel -j4 RUN_COMMANDS ::: “${FILES[@]}” | tee output.txt

You will want parallel -k ... if the order is important.

edited Nov 25, 2017 at 8:55

answered Nov 25, 2017 at 8:11

Mark Setchell

210k32 gold badges310 silver badges505 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Mark Setchell Over a year ago

Oops, add a space here RUN_COMMANDS() {

John Doe Over a year ago

Sorry this was a syntax error in my pseudo code , the actual code was fine. I still get this error

Mark Setchell Over a year ago

I presume you really are using bash?

Mark Setchell Over a year ago

If you are still getting errors, please click edit under your original question and paste in your actual code and the actual errors so folks can assist you further.

Mark Setchell Over a year ago

You need export -f check_ping somewhere a few lines before parallel ...

|

Ole Tange · Accepted Answer · 2017-11-27 01:07:47Z

1

#!/bin/bash

get_racks(){
    # query that outputs a list of racks
    echo rack01
}
export -f get_racks

get_hosts() {
    # query that outputs a list of hosts by rack(passed as param to the func)
    echo host01
    echo host02
}
export -f get_hosts

get_gw() {
    # query that outputs a list of gateways
    echo gw1
    echo gw2
}
export -f get_gw

ping_gw() {
    rack="$1"
    get_hosts "$rack" | parallel -S - --onall ping -c 5 ::: "`get_gw`"
}
export -f ping_gw
get_racks | parallel ping_gw && echo All hosts and gws are up

If you have more functions/variables/aliases then consider using env_parallel instead. That way env_parallel will do the exporting for you.

edited Nov 27, 2017 at 1:07

answered Nov 26, 2017 at 18:20

Ole Tange

34.2k9 gold badges93 silver badges112 bronze badges

4 Comments

John Doe Over a year ago

thanks for the answer @OleTange , however I get /bin/bash: ping gw : command not found . I read in another thread that you said it could be due to a bug with the version of parallel ?

Ole Tange Over a year ago

Are you copying the full text? I have just tested on all releases of GNU Parallel since 2014. They all work. It is unlikely to be a bug in GNU Parallel.

John Doe Over a year ago

Yes, still returns command not found . I'm running it as a regular user account.

Ole Tange Over a year ago

Most likely something is messed up in your environment. See if you can reproduce the error on any of these: sourceforge.net/projects/virtualboximage/files

ntj · Accepted Answer · 2017-11-25 08:24:02Z

0

Parallel can be used in a very similar way to xargs, it even has the params of xargs implemented.

#!/bin/bash

RUN_COMMANDS(){
    ...... 
}

FILES=(file1 file2 file3)

echo ${FILES[@]} | parallel RUN_COMMANDS | tee output.txt

answered Nov 25, 2017 at 8:24

ntj

17112 bronze badges

Collectives™ on Stack Overflow

Executing a for loop with gnu parallel

3 Answers 3

7 Comments

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related