How to execute multiple commands with concurrency and wait limit?

Question

What I would like to achieve is something like this:

#!/bin/sh
concurrency_limit 3

#takes 5 min
(/usr/bin/my-process-1 --args1 && /usr/bin/my-process-2 --args1) & 
#takes 10 min
(/usr/bin/my-process-1 --args2 && /usr/bin/my-process-2 --args2) &
#takes 15 min
(/usr/bin/my-process-1 --args3 && /usr/bin/my-process-2 --args3) &
#takes 5 min
(/usr/bin/my-process-1 --args4 && /usr/bin/my-process-2 --args4) &
#takes 10 min
(/usr/bin/my-process-1 --args5 && /usr/bin/my-process-2 --args5) &
#takes 20 min
(/usr/bin/my-process-1 --args6 && /usr/bin/my-process-2 --args6) &

wait max_limit 1200
echo all processes complete

Overall expected maximum execution time is 20 min (-+ 1min) and let's assume I have 3 cpu cores available and I don't want to have more than 3 processes running at the same time.

At the beginning of the script first 3 process started.

After 5 min: 1st process finished and 4th process started.

10th min: 2nd and 4th processes are finished and 5th process started.

15th min: 3rd process is finished.

20th min: 5th process is finished. 6th process is killed without further waiting.

I did a lot of research in stackoverflow but I couldn't find similar usage case:

How to wait in bash for several subprocesses to finish and return exit code !=0 when any subprocess ends with code !=0?

https://www.codeword.xyz/2015/09/02/three-ways-to-script-processes-in-parallel/

http://www.gnu.org/software/parallel/

Any help or comment would be appreciated thanks.

Mark Setchell · Accepted Answer · 2016-12-02 13:18:49Z

2

Unless I have missed something, I think GNU Parallel will do that for you quite readily.

If you make a file called jobs containing:

./my-process-1 --args1 && ./my-process-2 --args1
./my-process-1 --args2 && ./my-process-2 --args2
./my-process-1 --args3 && ./my-process-2 --args3
./my-process-1 --args4 && ./my-process-2 --args4
./my-process-1 --args5 && ./my-process-2 --args5
./my-process-1 --args6 && ./my-process-2 --args6

Then you can see what GNU Parallel will do using --dry-run as follows:

parallel --dry-run -j 3 -k -a jobs

Output

./my-process-1 --args1 && ./my-process-2 --args1
./my-process-1 --args2 && ./my-process-2 --args2
./my-process-1 --args3 && ./my-process-2 --args3
./my-process-1 --args4 && ./my-process-2 --args4
./my-process-1 --args5 && ./my-process-2 --args5
./my-process-1 --args6 && ./my-process-2 --args6

If my-process-1 takes 3 seconds and my-process-2 takes 5 seconds, the entirety requires 16 seconds, since the first 3 lines are executed in parallel and each line takes 8 seconds, then the next 3 lines are executed in parallel and take a further 8 seconds.

answered Dec 2, 2016 at 13:18

Mark Setchell

210k32 gold badges310 silver badges504 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mirza Over a year ago

thank you for the answer. but how about total time limit of execution ?

Mark Setchell Over a year ago

timeout 120 parallel ...

arturro · Accepted Answer · 2016-11-29 21:15:29Z

1

You can do it with xargs. For example below will run function "func" 6 times for arguments 3,3,4,1,4 and 15 using 3 parallel processes, and kill it after 10 secs:

function func  { echo args:$1; sleep $1; echo done; }
export -f func

function worktodo { echo -e 3\\n 3\\n 4\\n 1\\n 4\\n 15 | xargs -P 3 -I {} sh -c 'func "$@"' _ {}; }
export -f worktodo

timeout 10 sh -c "worktodo" || echo "timeout"

answered Nov 29, 2016 at 21:15

arturro

1,6061 gold badge10 silver badges13 bronze badges

Comments

gilez · Accepted Answer · 2016-11-29 21:42:20Z

Here's a skeleton, using SIGINT to communicate between the parent and your sub-processes.

Set a trap that counts how many processes are busy, and when one ends, start another:

trap '{ let Trapped++; }' INT  # start another child

Initialize that to how many you want to run in parallel:

Trapped=$ATONCE  # 3 in your case

Then loop and start children as needed:

while true
do
  # Assuming there's more work to do. You need to decide when to terminate
  do_work &

  while [ $Trapped -le 0 ]
      wait         # race condition, interruptible by SIGINT
      local rc=$?  # ...
  done
done

Then in do_work you need something like:

call-external-process with parms

# Deal with problems
[[ $? -ne 0 ]] && { .... }

# Now tell parent we're done
kill -INT $$

That's a crude idea. Missing is how you know when you have no more processes to start, and it needs better error handling, but hopefully you get the idea. There will be 3 processes running at all times, a new one started when one ends, until there's nothing left to do.

Collectives™ on Stack Overflow

How to execute multiple commands with concurrency and wait limit?

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related