Using Python subprocess to run SLURM script to submit multiple long jobs to queue and waiting for jobs to finish before continuing python script

Question

Basics of the situation are that I'm using Python's subprocess module to run a SLURM script which submits a number of jobs to a queue on some HPC resources that I use. I would like the Python script to wait until all the jobs submitted are finished. I tried doing this with:

proc = subprocess.Popen(['sbatch slurm.sh'],shell=True)
proc.wait()

However, it only waits for the jobs to be submitted and not for all the jobs to finish. Anyone have any suggestions for how to make this work?

Note: One idea I had was potentially having subprocess communicate somehow with the .out file produced by the SLURM script (and wait for that to finish being edited maybe?), but I'm not sure how that would work.

Thanks for any ideas!

jo_Veera · Accepted Answer · 2019-07-31 14:08:54Z

2

I have recently worked on slurm and have used -W or --wait parameter of sbatch to make the sbatch command first complete its execution and then execute the python code. Something like this sbatch -W slurm.sh

answered Jul 31, 2019 at 14:08

jo_Veera

3134 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

damienfrancois · Accepted Answer · 2023-08-18 16:23:44Z

1

Use a python library to interface with Slurm, such as

and you will be able to manage the slurm jobs in a very easy way.

edited Aug 18, 2023 at 16:23

answered Apr 28, 2017 at 13:34

damienfrancois

60.4k9 gold badges116 silver badges128 bronze badges

1 Comment

bmurph Over a year ago

The link to joblib tries to download a file immediately... maybe it should be removed or updated?

Collectives™ on Stack Overflow

Using Python subprocess to run SLURM script to submit multiple long jobs to queue and waiting for jobs to finish before continuing python script

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related