How to run several python files using bash script iteratively using only 7 CPUS?

Question

On my machine, which has eight CPUs, I wish to run a number of Python files. I only want to use 7 CPUs concurrently. Is it possible to automate this process?

Currently, I am doing

pids0=""
for i in $(seq 1.5 0.05 1.8); do
   python3 $i &
   pids0="$pids0 $!"
done
wait $pids0


pids1=""
for i in $(seq 1.85 0.05 2.0); do
   python3 $i &
   pids1="$pids1 $!"
done
wait $pids1

Each loop is designed in way to submit only 7 scripts, thus I have to write several for-loops to perform 100 calculations. As you can see, my approach is incredibly inefficient because I have to wait for 7 submitted python scripts to finish running before I can proceed to next loop.

My goal is to write code that can submit the next task as soon as I have any free CPU available after finishing the previous one without writing several for-loops.

PS: I am using my office computer and I am not an administrator.

bugless-soup · Accepted Answer · 2022-08-02 20:29:39Z

3

You can use GNU Parallel:

parallel -j 7 python3 script.py ::: $(seq 1.5 0.05 2.0)

EDIT according to Mark Setchell's comment

edited Aug 2, 2022 at 20:29

answered Aug 2, 2022 at 20:18

bugless-soup

713 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Mark Setchell Over a year ago

You need to replace -- arg1 arg2 arg3 with ::: $(seq 1.5 0.05 2.0)

Raghvender Over a year ago

Thanks @Mark, but as I have said I don't have admin rights, so cannot install gnu parallel.

Mark Setchell Over a year ago

@Raghvender Admin rights aren't needed - see this answer from Ole Tange (the author) stackoverflow.com/a/42396008/2836621

bugless-soup Over a year ago

@Raghvender Someone has already asked a similar question stackoverflow.com/questions/6441509/how-to-write-a-process-pool-bash-shell

Ole Tange Over a year ago

While you certainly can do ::: $(seq 1.5 0.05 2.0) I would personally do: seq 1.5 0.05 2.0 | parallel -j -1 python3 script.py. It makes no difference here, but if your next machine has 16 cores, it will use 16-1 of those. It will also work if the input parameters is longer than the longest line (e.g. seq 1.5 0.0000005 2.0)

Collectives™ on Stack Overflow

How to run several python files using bash script iteratively using only 7 CPUS?

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related