Better way of running a series of commands simultaneously in UNIX/Linux Shell?

Question

I want to know the good practice of performing a series of commands simultaneously in UNIX/Linux. Suppose that I have a program, program_a, which requires one parameter. I have stored parameters line by line in a file. So I wrote:

while read line
do
    ./program_a line > $line.log 2>&1
done < parameter_file

The problem is that execution of program_a takes long time. Because each executions of program_a for each parameter is independent, So I think these executions can be run simultaneously. I don't know if it regards to multithreading or other technique. The following is my thought. Use & to run each executions on the background.

while read line
do
    ./program_a line $line.log 2>&1 &
done < parameter_file

Is there any better way of launching multiple tasks?

That's the standard way to do it. You can use the wait command to hold up proceedings until all the child processes are complete. There are programs like parallel that you could use too. — Jonathan Leffler
– Jonathan Leffler, Commented Dec 2, 2013 at 11:22
What do you mean by 'better' exactly (ie what properties might a 'better' way have)? This is a normal and useful technique, is it giving you issues? — tolanj
– tolanj, Commented Dec 2, 2013 at 11:22
I believe you want to >> line.log instead of > line.log (overwriting previously written infos) ? and $line instead of line — Olivier Dulac
– Olivier Dulac, Commented Dec 2, 2013 at 11:35

UltraInstinct · Accepted Answer · 2013-12-02 11:48:52Z

5

Did you know that xargs can launch tasks in parallel? Check out -P -n parameters!

An example:

xargs -P 4 -n 1 ./program_a < parameter_file

That will start up to 4 (P=4) program_a instances for processing each line (n=1). You'll probably have to wrap program_a within a shell script or something so that child processes stdout & stderr can be redirected appropriately.

How this is better than putting processes to backgroud: Suppose you have 1000 lines in the input file, obviously you wouldn't want 1000 processes to be launched. Xargs allows you to look at it as a queue, with P workers each consuming and processing n items from it.

edited Dec 2, 2013 at 11:48

answered Dec 2, 2013 at 11:36

UltraInstinct

44.6k12 gold badges86 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ole Tange · Accepted Answer · 2013-12-02 23:34:30Z

1

With GNU Parallel you can get a logfile for each parameter and run one job per CPU core:

parallel --results logdir ./program_a :::: parameter_file

Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). You command line with love you for it.

answered Dec 2, 2013 at 23:34

Ole Tange

34.2k9 gold badges93 silver badges112 bronze badges

Collectives™ on Stack Overflow

Better way of running a series of commands simultaneously in UNIX/Linux Shell?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related