How to install python package/module in slurm

Question

This might be a very basic question, but this is my first time working with a slurm-cluster, and I don't want to mess anything up (the administrator is on vacation).

I have a python script that uses "import torch". When I run "sbatch myscript.sh", I get the output "ImportError: No module named 'torch'". I used pip inside the node to download the torch package, but after download I still get the importError.

How do I make my imports work? Should I download the sourcecode for the packages from github and upload them to my home directory on slurm? The guide I was given by the administrator didn't include information for this scenario, what am I missing?

You need to be sure the python found when calling myscript.sh is the same python as is called when using pip. — Open AI - Opting Out
– Open AI - Opting Out, Commented Apr 27, 2019 at 17:20

Shayan · Accepted Answer · 2019-12-18 02:50:47Z

10

You should import all these packages in your bash code instead. Here is an example:

#!/bin/bash
#SBATCH --account=def-someuser
#SBATCH --mem-per-cpu=1.5G      # increase as needed
#SBATCH --time=1:00:00

module load python/3.6
virtualenv --no-download $SLURM_TMPDIR/env
source $SLURM_TMPDIR/env/bin/activate
pip install --no-index --upgrade pip

pip install --no-index -r requirements.txt
python ...

You can put all dependencies inside the requirements file and install them all at once in the virtualenv. For more information take a look at this page.

answered Dec 18, 2019 at 2:50

Shayan

1703 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Teepeemm Over a year ago

I found that I only needed to pip install once, and not have that in my slurm script. When I activated the virtual environment, the packages were still installed and ready to go.

Shayan Over a year ago

@Teepeemm, yes, but what if each time you are connected to a new instance or you need to create multiple instances to do your task concurrently? That's why I put it in the slurm script.

Collectives™ on Stack Overflow

How to install python package/module in slurm

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related