How does one Parallelize a shell script with arguments using GNU parallel?

Question

I am new to bash scripting. I have a shell script that runs several functions for longitudinal image processing in Matlab through terminal. I would like to parallelize the process in the terminal.

Here is a brief example of how it runs:

./script.sh *.nii -surface -m /Applications/MATLAB_R2018b.app/bin/matlab

*.nii refers to images from a single subject taken at different times (i.e. subj1img1 subj1img2 subj3img3). There are 3 images per subject in my case. So in each run, the script runs through all images of a single subject.

I would like to parallelize this process so that I can run this script for multiple subjects at the same time. Reading through GNU parallel with my little experience I wasn't able to figure out the code I need to write to make it happen. I'd really appreciate if anyone has any suggestions.

Welcome to SO. Update your question to meet the standard of a Minimal, Complete, and Verifiable example. Mechanical Turk detection actived. SO is not a "we do your coding" for you site. So please show effort on what you tried to code and what failed. End of Triage Review. .. and finish the tour ;-) — ZF007
– ZF007, Commented Feb 19, 2019 at 1:21

Ole Tange · Accepted Answer · 2019-02-21 18:06:18Z

1

parallel ./script.sh {} -surface -m /Applications/MATLAB_R2018b.app/bin/matlab ::: *.nii

answered Feb 21, 2019 at 18:06

Ole Tange

34.2k9 gold badges93 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Amirhossein Modabbernia Over a year ago

Hi and thanks. so I tried this and given that I want to have a pair of files in each iteration I used two {} {}. But it seems like despite me defining the max number of jobs for parallel as 3, it opens MATLAB not 3 times but as many times as the number of file pairs (10 in this case). Not sure what to do with that because it will overload my system. Is there anyway to tell parallel to exactly have 3 jobs at any time? This is my code: parallel --jobs 3 ./cat_batch_long.sh {} {} -surface -m /Applications/MATLAB_R2018b.app/bin/matlab ::: *.nii

nullPointer · Accepted Answer · 2019-02-19 08:25:46Z

0

you can start them in the background using & in a for loop as below :

for f in *.nii
do
  ./script.sh "$f" -surface -m /Applications/MATLAB_R2018b.app/bin/matlab &
done

answered Feb 19, 2019 at 8:25

nullPointer

4,5891 gold badge20 silver badges31 bronze badges

Collectives™ on Stack Overflow

How does one Parallelize a shell script with arguments using GNU parallel?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related