How to flush GPU memory using CUDA on WSL2

Question

I have interrupted the training of the model in PyTorch on CUDA, which I've run on Windows Subsystem for Linux 2 (WSL2). The dedicated GPU memory of NVIDIA GeForce RTX 3080Ti was not flushed.

What I have tried:

gc.collect() and torch.cuda.empty_cache() does not resolve the problem (reference)
When running numba.cuda.select_device(0) to potentially cuda.close(), the notebook hangs (reference)
After running nvidia-smi to potentially reset the GPU (reference), the command prompt hangs
Win + Ctrl + Shift + B to reset the graphics stack in Windows does not help (reference)
Restarting the notebook kernel as well as restarting the notebook server does not help
Physical reset is not available

UPDATE:

Running nvidia-smi in the command prompt on Windows (not on WSL2) yields the following

Lorewalker Cho · Accepted Answer · 2022-08-22 17:46:05Z

1

I don't know your actual environment.

suppose that you use anaconda window-venv

On cmd >nvidia-smi shows following.

Check pid of python process name(>envs\psychopy\python.exe).
On cmd taskkill /f /PID xxxx

this could be help.

and you don't want doing like this.

if you feeling annoying you can run the script on prompt, it would be automatically flushing gpu memory.

answered Aug 22, 2022 at 17:46

Lorewalker Cho

707 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Python Over a year ago

I ran nvidia-smi in the Windows cmd. A screenshot is in the updated question. I don't find \psychopy\python.exe, though nor do I see any similar processes, which it would make sense to kill. Do you see any?

Lorewalker Cho Over a year ago

I use windows only, i'm always see .\python.exe but your env not showing. how about command run on wsl environment

Lorewalker Cho Over a year ago

I think you are not using gpu in pytorch similar problem is in here. stackoverflow.com/questions/15197286/… which hold your gpu memory check first

Python Over a year ago

As I mentioned in the description, when running nvidia-smi on WSL2, the command prompt freezes, so I don't get any info about it. Nevertheless, I have managed to physically reboot the machine. Problem is solved. Thanks

Lorewalker Cho Over a year ago

you getting hard time. so sorry. try 'nvidia-smi -r' or 'nvidia-smi --gpu-reset' may be this is majino line.

Collectives™ on Stack Overflow

How to flush GPU memory using CUDA on WSL2

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related