How to get maximum GPU memory usage for a process on Ubuntu? (For Nvidia GPU)

Question

I have a server with Ubuntu 16.04 installed. It has a K80 GPU. Multiple processes are using the GPU.

Some processes have unpredictable GPU usage, and I want to reliably monitor their GPU usage.

I know that you can query GPU usage via: nvidia-smi, but that only gives you the usage at the queried time.

Currently I query the information every 100 ms, but that's just sampling the GPU usage, and can potentially skip peak GPU usage.

Is there a reliable way for me to get the maximum GPU memory usage for a given PID process?

You probably want to intercept calls to cuMemAlloc(), cuMemFree () and related functions via a library inserted with the LD_PRELOAD trick. Then you can record the peak/sustained/any-metric-you-want memory use in any way you want. — tera
– tera, Commented Mar 14, 2017 at 19:27

KuroNeko · Accepted Answer · 2017-03-16 03:54:27Z

1

Try using the NVIDIA Visual Profiler. I am not sure how accurate it is but it gives you a graph of the device memory usage at different times when your program is running.

answered Mar 16, 2017 at 3:54

KuroNeko

3312 gold badges8 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to get maximum GPU memory usage for a process on Ubuntu? (For Nvidia GPU)

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related