How to activate a specific Python environment as part of my submission to Slurm?

Question

I want to run a script on cluster (SBATCH file).

How can activate my virtual environment (path/to/env_name/bin/activate).
Does I need only to add the following code to my_script.sh file?

module load python/2.7.14 source "/pathto/Python_directory/ENV2.7_new/bin/activate"

Just run the python executable in the bin folder of the virtual environment. — Klaus D.
– Klaus D., Commented Nov 29, 2018 at 18:50
I think @KlausD. is suggesting you run something like /path/to/miniconda3/envs/allennlp/bin/python -m allennlp.run --version. — schmmd
– schmmd, Commented Aug 27, 2019 at 15:38
Would you mind posting a full example of the solution, please? — Kaleb Coberly
– Kaleb Coberly, Commented Feb 9, 2023 at 2:53

Trond Kristiansen · Accepted Answer · 2018-11-29 19:01:31Z

24

You mean to activate a specific Python environment as part of your submission to Slurm? This is what I add to my job script and it works well. Note that I use Anaconda, which by default adds the required paths to my .bashrc script after installation. Hope this helps.

....
# define and create a unique scratch directory
SCRATCH_DIRECTORY=/global/work/${USER}/kelp/${SLURM_JOBID}
mkdir -p ${SCRATCH_DIRECTORY}
cd ${SCRATCH_DIRECTORY}

# Activate Anaconda work environment for OpenDrift
source /home/${USER}/.bashrc
source activate MyEnvironment 

# we execute the job and time it
time mpirun python slurmscript.py

answered Nov 29, 2018 at 19:01

Trond Kristiansen

2,46625 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Charlie Parker Over a year ago

what if my submission script is a python script?

Trond Kristiansen Over a year ago

You should be able to just execute your python script where I call mpirun. At that point the Anaconda environment is setup

damienfrancois · Accepted Answer · 2018-11-30 14:55:11Z

3

Yes. Just make sure to insert those lines after the SBATCH ... lines and before any use of python.

answered Nov 30, 2018 at 14:55

damienfrancois

60.5k9 gold badges116 silver badges128 bronze badges

3 Comments

Charlie Parker Over a year ago

what if my submission script is a python script? I can't just do that...

rohit_r Over a year ago

how do i create this environment on ALL nodes of my slurm cluster?

Kaleb Coberly Over a year ago

Which lines? Can you post a full example?

Collectives™ on Stack Overflow

How to activate a specific Python environment as part of my submission to Slurm?

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related