run a bash for loop in parallel

Question

I've got this script that does a credential lookup for each host, in an on-premise vault system, and then runs an ansible-playbook for it.

#!/bin/bash

for host in `cat ~/.ansible/hosts`
  do
    SECRET=`/opt/vault/bin/get-admin-credential --tag=$host`
    HOST=`echo $SECRET | cut -d ';' -f1`
    LOGIN=`echo $SECRET | cut -d ';' -f2`
    DOMAIN=`echo $SECRET | cut -d ';' -f3`
    PWD=`echo $SECRET | cut -d ';' -f4`

    if [ -z "$DOMAIN" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
    else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
    fi
  done

This loops over each host sequentially, I've tried stuff with GNU parallel but haven't been able to do what I want, running the for loop with 5 in parallel.

Anyone point me in the right direction?

Not familiar with ansible but (this may be a dumb question), have you tried creating a function? e.g., function parallel_fn { for ... do ... done } followed by parallel_fn & parallel_fn? — C. Dunn
– C. Dunn, Commented Mar 25, 2020 at 13:52
@C.Dunn I don't think that will run the 5 ansible processes in parallel with each other. Rather, it will create a loop that runs the 5 ansible processes sequentially one after the other but which runs in parallel alongside the caller of the function. — Mark Setchell
– Mark Setchell, Commented Mar 25, 2020 at 14:46

Mark Setchell · Accepted Answer · 2020-03-25 14:29:38Z

2

I don't have any "ansibles" or "vaults", so this is completely untested but may get you close:

doit(){
   host="$1"

   SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
   HOST=$(echo $SECRET | cut -d ';' -f1)
   LOGIN=$(echo $SECRET | cut -d ';' -f2)
   DOMAIN=$(echo $SECRET | cut -d ';' -f3)
   PWD=$(echo $SECRET | cut -d ';' -f4)

   if [ -z "$DOMAIN" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
   else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
   fi
}

# Export doit function to subshells created by GNU Parallel
export -f doit

parallel -a ~/.ansible/hosts doit

Stylistically, there are maybe a few improvements. Firstly, shell variables consisting of upper case letters are reserved, so you shouldn't maybe use HOST, DOMAIN etc. Also, you can probably simplify all that unsightly cutting and echoing to extract the variables from the SECRET by using an IFS=';' and a read like this:

SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
IFS=';' read host login domain pwd <<< "$SECRET"

So, my best and final answer is:

doit(){
   host="$1"

   secret=$(/opt/vault/bin/get-admin-credential --tag=$host)
   IFS=';' read host login domain pwd <<< "$secret"

   if [ -z "$domain" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login ansible_password=$pwd" --limit $host
   else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login@$domain ansible_password=$pwd" --limit $host
   fi
}

# Export doit function to subshells created by GNU Parallel
export -f doit

parallel -a ~/.ansible/hosts doit

edited Mar 25, 2020 at 14:29

answered Mar 25, 2020 at 14:24

Mark Setchell

210k32 gold badges310 silver badges504 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Greg Over a year ago

Thanks! I've changed the --limit $host into --limit $1. My for loop was a bit confusing with both for host in and then also the host variable. Not really necessary in this case, but is it possible to combine the input source parameter with --jobs to limit the simultaneous runs? No matter what amount I specify it does the same with or without the parameter.

Mark Setchell Over a year ago

There are several useful enhancements that you could port across from @chepner answer... the if statement, the -r and trailing _ on the read statement...

Mark Setchell Over a year ago

You should be able to use parallel -j 4 ... to do just 4 at a time.

chepner · Accepted Answer · 2020-03-25 14:40:09Z

1

You simply need to run ansible-playbook in the background using the & command terminator. Note, though, that the entire loop can be simplified and improved.

run_playbook () {
  ansible-playbook -i ~/.ansible/hosts \
                   -e "ansible_host=$2 ansible_login=$3 ansible_password=$4" \
                   ~/.ansible/windows.yml --limit "$1"
}

while IFS= read -r host; do
    secret=$(/opt/vault/bin/get-admin-credential --tag="$host")
    IFS=";" read -r shost slogin sdomain spasswd _ <<< "$secret"
    if [[ -n $sdomain ]]; then
      login="$slogin@$sdomain"
    fi
    run_playbook "$host" "$shost" "$login" "$password" &
done < ~/.ansible/hosts

answered Mar 25, 2020 at 14:40

chepner

538k77 gold badges594 silver badges746 bronze badges

1 Comment

rivimey Over a year ago

While the code here is better in many ways, it will run all the hosts it can at once, not just the 5 (at a time) requested. I would also suggest that the ansible log output was saved to a per host log file, as as things stand all output will be intermingled very confusingly.

Collectives™ on Stack Overflow

run a bash for loop in parallel

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related