I have several different perl scripts that I am running simultaneously. All of them take the same text file as input (textfile.txt). Is there a way that I could write the file name once instead of writing it several times?
$ perl myscript1.pl textfile.txt & perl myscript2.pl textfile.txt &
perl myscript3.pl textfile.txt & perl myscript4.pl textfile.txt &
perl myscript5.pl textfile.txt & perl myscript6.pl textfile.txt &
perl myscript7.pl textfile.txt & perl myscript8.pl textfile.txt
With GNU Parallel
parallel perl {} textfile.txt ::: myscript*.pl
Run with parallel --dry-run ... first if you want to see what it would do, but without actually doing anything.
Note that if you have 4 CPU cores, it will start 4 jobs in parallel and then start a new one each time a job finishes, and that is subtly different from having all running at once. If you have, say 32 jobs but only 4 cores and all 32 should be running at once, use parallel -j 32 ... for example.
1. If I have 50 CPU cores and 100 jobs, but only want to run 32 jobs at once, will parallel -j 32 achieve that for me? 2. What would I change in the code above if I had more text files and I wanted all of them to be passed to all of my perl scripts?
parallel perl {} p1 p2 p3 p4 p5 p6 ::: script*.plall of the text files (e.g. first one generated is perl script1.pl p1.txt p2.txt p3.txt p4.txt p5.txt p6.txt. But I realized that replacing {} with ::: instead works! But I have to also exchange the place of scripts with the text files, therefore: parallel --dry-run perl ::: script*.pl ::: text1.txt text2.txt text3.txt text4.txt. Does that make any sense?run_all_perl() (
perl_scripts=(
myscript1.pl myscript2.pl myscript3.pl myscript4.pl
myscript5.pl myscript6.pl myscript7.pl myscript8.pl
)
for script in "${perl_scripts[@]}"; do
perl "$script" "$1" &
done
wait
)
run_all_perl textfile.txt
