How to use gnu_parallel to run multiple executable and/or bash scripts?

Question

I've been recently attempting to run my scripts in parallel in a more convenient way than to open a several instances of terminal and executing in scripts separately.

I've been trying to learn how to use gnu_parallel for the past couple of days and I am still a bit clueless, and hoping if someone can provide a direct example.

Suppose I have a g++ compiled code called blah.exe and a bash script called blah.sh that will run alone perfectly fine, but I want to execute them in different directories.

I've been reading https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Working-as-xargs--n1.-Argument-appending

and

https://www.biostars.org/p/182136/

but I am not totally clear about the syntax

To run these in series, I would do:

for i in 1 2 3 4
mv ./blah.exe directory$i
cd directory$i
./blah.exe all
cd ..
end

similarly

for i in 1 2 3 4
mv ./blah.sh directory$i
cd directory$i
source ./blah.sh all
cd ..
end

I am trying to under stand how I would split this load to 4 logical-threads in one command using parallel.

Could someone provide an example for this?

Thank you for your time.

does your script affect any shared ressource that could make the parallelization somehow harder ? — Reda Meskali
– Reda Meskali, Commented Apr 29, 2019 at 14:36
If you want to run a process in the background in bash, then append & to the command. See also en.wikipedia.org/wiki/Job_control_(Unix) — Jesper Juhl
– Jesper Juhl, Commented Apr 29, 2019 at 14:43
Jesper, yes I can use bash and &, but I want to write a single script to parallelize using gno_parallel — Sj L
– Sj L, Commented Apr 29, 2019 at 14:59
Reda, as far as I know, I carefully mallocated and won't be extra hard — Sj L
– Sj L, Commented Apr 29, 2019 at 15:01

Mark Setchell · Accepted Answer · 2019-04-29 16:01:31Z

2

Something like:

parallel --dry-run 'cd directory{}; ../blah.exe all; source ../blah.sh all' ::: {1..4}

No need to copy/move the executable, just run the same one.

No need to cd .. afterwards, as it's a new process each time.

Note this is not multi-threading, it is multi-processing.

If you want to process discontiguous directory numbers, you can use:

parallel ... ::: {1..4} 6 7 {11..14}

If you want to process all directories, you can use:

printf "%s\0" */ | parallel -0 'cd {}; pwd'

If you want to process all directories starting with FRED, you can use:

printf "%s\0" FRED*/ | parallel -0 'cd {}; pwd'

edited Apr 29, 2019 at 16:01

answered Apr 29, 2019 at 15:06

Mark Setchell

210k32 gold badges310 silver badges505 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Sj L Over a year ago

Thank you Mark. You are right, this is multi-processing, sorry for the misclassification, I will change that. This is pretty much it. I only have one more questions for you: First, say if I wanted to do 1..4 then 6, the 11..20, what would be the proper syntax? I tried running with 1,2,3,4,6, ... Another is is there an option to print the printouts and logs?

Mark Setchell Over a year ago

I have added an example of the syntax for discontiguous directory numbers. Can you clarify or question about the logs and printouts please? Which logs and printouts? Where are they?

Sj L Over a year ago

Regarding the log, please nevermind. It got saved in the cache somehow, from testing I found a way to just use "command >log.log"

Sj L Over a year ago

However, I think this may be against the policy of one topic per question but I have even one more question. Suppose that I have a huge list of files that I need to execute "blah.exe" on. Say I have the file name list in "glah.tmp" I know to use while command <<glah.tmp is there a gnu_parallel equivalent to this?

Mark Setchell Over a year ago

Questions are free - and answers! I think it might be better to ask another one as there are potentially several solutions and it could get messy. You can always click share under your original question and copy the hyperlink and paste it into the new question for reference. I'll keep an eye out for a new question, and I am sure Ole Tange (the author of parallel) will see it too, so you should get a response quite soon.

Collectives™ on Stack Overflow

How to use gnu_parallel to run multiple executable and/or bash scripts?

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related