Remove GPU Ram with VLLM

Question

After running this code within a jupyter notebook, it runs properly. However, the memory is still stored in the GPU. How do I get rid of this memory to clear up space on my GPU. Sorry if I am formatting this question poorly, I am not used to posting. Provided is the code:

llm = LLM(
  model=model_path, 
  gpu_memory_utilization=0.7, 
  max_model_len=2048,
)

llm = LLM(model=model_path, dtype=torch.bfloat16, trust_remote_code=True, max_model_len=2048, quantization="bitsandbytes", load_format="bitsandbytes", gpu_memory_utilization = 0.8)

I tried deleting llm and clearing cache which decreases the allocated and chached memory, but I cannot rerun the LLM method as I get an OOM Error (the previous call still has stored memory).

Please refer to this issue and try: github.com/vllm-project/vllm/issues/1908. If you find it works, consider writing an answer down below. — MinhNH
– MinhNH, Commented Feb 14 at 10:46

heyzude · Accepted Answer · 2025-02-14 16:45:06Z

0

Well, how about killing the vllm related process using pkill -9 -ef <part or whole of the vllm process name or cli command>? You can check the vllm process consuming GPU RAM with nvidia-smi, nvitop or nvtop.

answered Feb 14 at 16:45

heyzude

3831 gold badge4 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Brandon Pardi · Accepted Answer · 2025-02-19 00:48:47Z

0

I regularly use the holy trinity of cleanup with pytorch

Delete model object with Python del
Empty cache with torch.cuda.empty_cache()
Python garbage collection gc.collect() (import gc at top of script)

del llm
torch.cuda.empty_cache()
gc.collect()

That said Jupyter notebooks are weird, you may just have to restart the kernel if these don't work since Jupyter has its own cachine mechanisms.

answered Feb 19 at 0:48

Brandon Pardi

2481 silver badge11 bronze badges

Collectives™ on Stack Overflow

Remove GPU Ram with VLLM

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related