1

I have thousands of remote machines. I want to run some specific commands on all of them, but in parallel.

I use these commands which run sequentially:

for f in `cat host.lst`
do
       ./runScript.sh $f
done

Let's suppose host.lst contains 100 hosts. I want to run runScript.sh on 100 hosts in parallel. Also logs should be maintained.

I can not install any utility on my machine such as PSSH.

I have done a lot of research and found these links, but they did not help. I do not understand how they work:

   Automatically run commands over SSH on many servers

   Execute command on multiple files matching a pattern in parallel

Can anyone explain the logic?

Improve this question
2
  • 1
    There are answers in both the questions that you referenced that don't require pssh. Some use GNU parallel (which would be my choice) but there are others that use only commands that are commonly installed on Unix-like systems. I'd advise that you edit the question to show which of those answers you've tried and detail the problems you encountered. As it stands, this question is a duplicate. Commented Apr 12, 2017 at 8:55
  • See if you understand gnu.org/software/parallel/… You do not need to be root to use GNU Parallel - even if it is not installed: oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html Commented May 4, 2017 at 22:35

2 Answers 2

Reset to default
6 
logdir=`mktemp -d`
bunch=200

IFS=$'\n'
for hosts in $(< hosts.lst xargs -r -L "$bunch"); do
   IFS=" ";
   for host in $hosts; do
      ssh -n -o BatchMode=yes "$host" './runScript.sh' 1>"$logdir/$host.log" 2>&1 &
   done
   wait
done

Assuming the 1000s (thousands) of hosts are listed one/line in the hosts.lst file and then from these a bunch are selected in one time (200), and on each of these 200 hosts are spawned your runScript.sh using ssh in batch mode and at the same time preserving the stdout+stderr spewing forth from each of these backgrounded job into a file with the name of host in the directory $logdir, which may be examined as and when required.

Finally we wait for one bunch to get over before we launch the next bunch, by means of the wait command at the end of the inner for loop.

Improve this answer
6
  • Rakesh, i think you were trying to write 1000s ???? u wrote 100s. Commented Apr 12, 2017 at 22:18
  • I just want to trigger the runscript only. so can " ssh -n -o Batch mode=yes "$host" " can be removed?? Commented Apr 12, 2017 at 22:21
  • what do you mean trigger runscript.sh? If you want to remve ssh then why bother about that hosts list ? Commented Apr 12, 2017 at 22:44
  • sorry "$host" is needed runscript.sh will take care of every thing like scp, rsync and all. I just need to trigger runscript.sh. Commented Apr 13, 2017 at 0:51
  • Thanks Rakesh, It worked !! last question, Can i use exec in place of xargs?? What would be the impact.. Commented Apr 13, 2017 at 4:31
2

The following modification to your script, which doesn't require any other utilities, will execute in parallel, but may run into limits depending on your hosts' limitations of open file handles:

for f in $(cat host.lst); do
       ./runScript.sh $f &
done

To capture the results to a log, just add > run.log after done to save to a new file, run.log.

The key change is the addition of &, which runs a process in the background rather than waiting for it to complete before executing the next command.

Improve this answer
3
  • If the count is high in list suppose 1000, then it should pick the 200 one time..Because changing the list of host again will lead to manual task. Commented Apr 11, 2017 at 23:35
  • you could use split to divvy up the list into batches, and do each batch in parallel. Commented Apr 12, 2017 at 16:25
  • in that case i have to maintain 10 for loop ( 10 batches). Commented Apr 12, 2017 at 22:15

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.