Skip to main content
deleted 8 characters in body
Source Link
aviro
  • 7k
  • 17
  • 36
xargs -I '{}' -P <max-procs> --arg-file <INPUTFILE> bash -c "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"

or

cat <INPUTFILE> | xargs -I '{}' -P <max-procs> bash -c "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"
exec    Executes an arbitrary command for each target host. The first of the pdsh remote arguments is the local command
        to execute, followed by any further arguments. Some simple parameters  are  substitued  on  the  command  line,
        including  %h  for  the target hostname, %u for the remote username, and %n for the remote rank [0-n] (To get a
        literal % use %%).  For example, the following would duplicate using the ssh module to run  hostname(1)  across
        the hosts foo[0-10]:

                  pdsh -R exec -w foo[0-10] ssh -x -l %u %h hostname

               and this command line would run grep(1) in parallel across the files console.foo[0-10]:

                  pdsh -R exec -w foo[0-10] grep BUG console.%h

-f number
       Set the maximum number of simultaneous remote commands to number.  The default is 32.
xargs -I '{}' -P <max-procs> --arg-file <INPUTFILE> "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"

or

cat <INPUTFILE> | xargs -I '{}' -P <max-procs> "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"
exec    Executes an arbitrary command for each target host. The first of the pdsh remote arguments is the local command
        to execute, followed by any further arguments. Some simple parameters  are  substitued  on  the  command  line,
        including  %h  for  the target hostname, %u for the remote username, and %n for the remote rank [0-n] (To get a
        literal % use %%).  For example, the following would duplicate using the ssh module to run  hostname(1)  across
        the hosts foo[0-10]:

                  pdsh -R exec -w foo[0-10] ssh -x -l %u %h hostname

               and this command line would run grep(1) in parallel across the files console.foo[0-10]:

                  pdsh -R exec -w foo[0-10] grep BUG console.%h

-f number
       Set the maximum number of simultaneous remote commands to number.  The default is 32.
xargs -I '{}' -P <max-procs> --arg-file <INPUTFILE> bash -c "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"

or

cat <INPUTFILE> | xargs -I '{}' -P <max-procs> bash -c "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"
exec    Executes an arbitrary command for each target host. The first of the pdsh remote arguments is the local command
        to execute, followed by any further arguments. Some simple parameters  are  substitued  on  the  command  line,
        including  %h  for  the target hostname, %u for the remote username, and %n for the remote rank [0-n] (To get a
        literal % use %%).  For example, the following would duplicate using the ssh module to run  hostname(1)  across
        the hosts foo[0-10]:

          pdsh -R exec -w foo[0-10] ssh -x -l %u %h hostname

       and this command line would run grep(1) in parallel across the files console.foo[0-10]:

          pdsh -R exec -w foo[0-10] grep BUG console.%h

-f number
       Set the maximum number of simultaneous remote commands to number.  The default is 32.
Source Link
aviro
  • 7k
  • 17
  • 36

I can offer two ways to do that.

xargs

Assuming you have a file that contains the list of hostnames separated by newlines, and that the user and port for all connections, you could use xargs.

xargs -I '{}' -P <max-procs> --arg-file <INPUTFILE> "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"

or

cat <INPUTFILE> | xargs -I '{}' -P <max-procs> "ssh -o StrictHostKeyChecking=no -p $connectivity_port $user@{} 'bash -s' < $file $scriptargs > $OUT_FOLDER/{}.log 2>&1"

You can set up concurrency with the -P flag.

       --max-procs=max-procs
       -P max-procs
              Run up to max-procs processes at a time; the default is  1.   If
              max-procs  is 0, xargs will run as many processes as possible at
              a time.  Use the -n option with -P; otherwise chances  are  that
              only one exec will be done.

It will write the output of each command to $OUT_FOLDER/$HOST.log.

If you have different user and port for each machine you can still use xargs, but that would be a bit more complex.

pdsh

Another option is to use pdsh which can "issue commands to groups of hosts in parallel".

pdsh -R exec -w^<INPUT FILE> -f <max-procs> bash -c "ssh -o StrictHostKeyChecking=no -p $connectivity_port %u@%h 'bash -s' < $file $scriptargs 2>&1"

The -f here is similar to the -P flag in xargs.

exec    Executes an arbitrary command for each target host. The first of the pdsh remote arguments is the local command
        to execute, followed by any further arguments. Some simple parameters  are  substitued  on  the  command  line,
        including  %h  for  the target hostname, %u for the remote username, and %n for the remote rank [0-n] (To get a
        literal % use %%).  For example, the following would duplicate using the ssh module to run  hostname(1)  across
        the hosts foo[0-10]:

                  pdsh -R exec -w foo[0-10] ssh -x -l %u %h hostname

               and this command line would run grep(1) in parallel across the files console.foo[0-10]:

                  pdsh -R exec -w foo[0-10] grep BUG console.%h

-f number
       Set the maximum number of simultaneous remote commands to number.  The default is 32.

If will dump the output of the commands prefixed with HOSTNAME:

Here's an example.

$ pdsh -R exec -w host1,host2 bash -c "ssh  -o StrictHostKeyChecking=no -p 22 %u@%h 'bash -s' <<< 'echo Running script on %h with arguments: \${@}' arg1 arg2 arg3"
host1: Running script on host1 with arguments: arg1 arg2 arg3
host2: Running script on host2 with arguments: arg1 arg2 arg3