How to feed a list of bash commands to xargs to run in parallel?

Question

I would like to run several instances of a program on a varying number of input file in parallel. The program itself is not parallellised, this is why I am looking for a way to submit multiple instances. I'm aware of GNU parallel, however the bash script that I'm writing will be shared with my co-workers, and not all of them have it installed.

I found an answer that almost matches my needs here, however the number of processes there are hardcoded, so I can't use a here document. In my case, there will be a different number of input files, so I thought I can list them and then feed to xargs to execute them. I tried various ways but neither of them worked. Two of my attemps to modify the code from the link:

#!/bin/bash
nprocs=3
# Attempt one: use a loop
commands=$( for ((i=0; i<5; i++)); do echo "sleep $i; echo $i;"; done )
echo Commands:
echo $commands
echo
{
    echo $commands | xargs -n 1 -P $nprocs -I {} sh -c 'eval "$1"' - {}
} &
echo "Waiting for commands to finish..."
wait $!

# Attempt two: use awk, the rest as above
commands=$( awk 'BEGIN{for (i=1; i<5; i++) { printf("sleep %d && echo \"ps %d\";\n", i, i) }}' )

The commands are executed one after the other. What could be wrong? Thanks.

choroba · Accepted Answer · 2019-07-30 08:26:52Z

2

Try running just

xargs -n 1

to see what commands are being run.

To avoid problems with quoting, I'd use an array of commands.

#! /bin/bash
nprocs=3

commands=()
for i in {0..4} ; do
    commands+=("sleep 1; echo $i")
done

echo Commands:
echo "${commands[@]}"

printf '%s\n' "${commands[@]}" \
| xargs -n 1 -P $nprocs -I % bash -c % &

echo "Waiting for commands to finish..."
wait $!

edited Jul 30, 2019 at 8:26

answered Jul 30, 2019 at 7:40

choroba

245k27 gold badges221 silver badges304 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Suzanka Over a year ago

Great, thanks! Could you also specify what the percent sign means?

choroba Over a year ago

I just use it instead of {} as it's shorter ;-)

Ole Tange · Accepted Answer · 2019-07-30 18:15:42Z

1

parallel --embed (version >20180122) is made for your situation:

parallel --embed > newscript.sh

Now edit the last lines of newscript.sh and you have GNU Parallel included in your script that you can distribute.

answered Jul 30, 2019 at 18:15

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Collectives™ on Stack Overflow

How to feed a list of bash commands to xargs to run in parallel?

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related