20

Is there a way in python 3 to log the memory (ram) usage, while some program is running?

Some background info. I run simulations on a hpc cluster using slurm, where I have to reserve some memory before submitting a job. I know that my job require a lot of memory, but I am not sure how much. So I was wondering if there is a simple solution for logging the memory over time.

Improve this question

3 Answers 3

Reset to default
11

You can do that with the memory_profiler package. Just with adding a decorator @profile to a function, you will get an output like this:

Line #    Mem usage  Increment   Line Contents
==============================================
 3                           @profile
 4      5.97 MB    0.00 MB   def my_func():
 5     13.61 MB    7.64 MB       a = [1] * (10 ** 6)
 6    166.20 MB  152.59 MB       b = [2] * (2 * 10 ** 7)
 7     13.61 MB -152.59 MB       del b
 8     13.61 MB    0.00 MB       return a

Otherwise, the easiest way to do it is to ask Slurm afterwards with the sacct -l -j <JobId> command (look for the MaxRSS column) so that you can adapt for further jobs.

Also, you can use the top command while running the program to get an idea of its memory consumption. Look for the RES column.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

This works perfect. Especially the 'mprof' command from memory_profiler is useful to get a memory over time plot.
3

You can use subprocess module. Here is a sample output of bash command free 

$ free -m
             total       used       free     shared    buffers     cached
Mem:          7979       7678        300          0        109       4628
-/+ buffers/cache:       2941       5038
Swap:         2046        360       1686

Python program - 

import subprocess
result = subprocess.check_output(['bash','-c', 'free -m'])
free_memory = result.split('\n')[1].split()[3]
# print free_memory
# 300

If you want to check memory usage of some process or periodically log it, then you can use pmap or some other utility depending on your use case and then parse the output.

Improve this answer

1 Comment

Thanks for your answer, if not for damienfrancois' one, I would have excepted it.
0

bash script tracking resources of python (or any other) process

Here some example script, which might come handy. This has the advantage, that you do not need to modify your python code. Disadvantage: you need a bash. Of course you can also run a python script from a file. Having the python code in the bash script is just for demonstration purposes.

script

#!/bash/bin
set -e

LOG_OUTPUT="output.log"
LOG_RESOURCES="resources.log"

PYTHON_CODE="
import numpy as np
from datetime import datetime
import time

arr = np.zeros(1000)

print('start', datetime.now(), flush=True)
for ii in range(5000000):  # work
    arr += 1
    
print('break start', datetime.now(), flush=True)
time.sleep(3)  # break
print('break end', datetime.now(), flush=True)

for ii in range(5000000):  # work
    arr += 1
print('end', datetime.now(), flush=True)
"

rm -f ${LOG_OUTPUT}
rm -f ${LOG_RESOURCES}

log_process () {
  while ps -p $1 > /dev/null
  do
    top -p $1 -b -d 1 -n 1 >> $LOG_RESOURCES
    printf "\n\n" >> $LOG_RESOURCES
    sleep 1.
  done
}

python -c "${PYTHON_CODE}" >> ${LOG_OUTPUT} 2>&1 &  # run python in background
PID_PYTHON=$!  # python process id
echo "PID_PYTHON=${PID_PYTHON}"
tail -f ${LOG_OUTPUT} &  # show output on terminal
log_process $PID_PYTHON  # log needed resources to file

terminal output

$ bash monitor_python_script.sh 
PID_PYTHON=6807
start 2023-05-04 10:08:56.510581
end 2023-05-04 10:09:01.770897
break start 2023-05-04 10:09:04.280212
break end 2023-05-04 10:09:07.283365
end 2023-05-04 10:09:14.875994

log file output: output.log

start 2023-05-04 10:08:56.510581
break start 2023-05-04 10:09:04.280212
break end 2023-05-04 10:09:07.283365
end 2023-05-04 10:09:14.875994

log file resources: resources.log

top - 10:08:56 up  1:47,  0 users,  load average: 0.08, 0.10, 0.09
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 16.1 us, 10.4 sy,  0.0 ni, 73.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29084.9 free,   3086.8 used,   2072.7 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30686.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:00.38 python


top - 10:08:57 up  1:47,  0 users,  load average: 0.08, 0.10, 0.09
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.6 us,  0.5 sy,  0.0 ni, 84.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.8 free,   3087.9 used,   2072.7 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30685.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:01.53 python


top - 10:08:58 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.2 us,  0.0 sy,  0.0 ni, 84.4 id,  0.5 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.9 free,   3087.8 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30685.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:02.69 python


top - 10:09:00 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.3 us,  0.0 sy,  0.0 ni, 85.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.1 free,   3088.5 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30684.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:03.85 python


top - 10:09:01 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.3 us,  0.0 sy,  0.0 ni, 85.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29083.3 free,   3088.3 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30685.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:05.01 python


top - 10:09:02 up  1:47,  0 users,  load average: 0.23, 0.13, 0.10
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  0.5 sy,  0.0 ni, 92.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.5 st
MiB Mem :  34244.4 total,  29096.4 free,   3075.2 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:06.17 python


top - 10:09:03 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  0.0 sy,  0.0 ni, 92.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29096.4 free,   3075.2 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:07.32 python


top - 10:09:04 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29096.9 free,   3074.7 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 S   0.0   0.1   0:08.05 python


top - 10:09:05 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.0 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29096.6 free,   3074.9 used,   2073.0 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30698.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 S   0.0   0.1   0:08.05 python


top - 10:09:07 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  0.5 sy,  0.0 ni, 98.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29087.0 free,   3084.6 used,   2072.8 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 S   0.0   0.1   0:08.05 python


top - 10:09:08 up  1:47,  0 users,  load average: 0.29, 0.14, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.6 us,  0.0 sy,  0.0 ni, 92.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.5 free,   3085.1 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:08.95 python


top - 10:09:09 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.1 us,  0.0 sy,  0.0 ni, 91.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.7 free,   3084.9 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.6 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:10.11 python


top - 10:09:10 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.0 us,  0.0 sy,  0.0 ni, 92.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29087.0 free,   3084.5 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:11.27 python


top - 10:09:11 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.1 us,  0.0 sy,  0.0 ni, 91.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.5 free,   3085.1 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:12.43 python


top - 10:09:12 up  1:47,  0 users,  load average: 0.35, 0.16, 0.11
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us,  0.0 sy,  0.0 ni, 92.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.3 free,   3085.3 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.2 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:13.59 python


top - 10:09:13 up  1:47,  0 users,  load average: 0.40, 0.17, 0.12
Tasks:   1 total,   1 running,   0 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8.1 us,  0.0 sy,  0.0 ni, 91.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  34244.4 total,  29086.2 free,   3085.3 used,   2072.9 buff/cache
MiB Swap:   1024.0 total,   1024.0 free,      0.0 used.  30688.1 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6807 user      20   0  532700  28592  14444 R 100.0   0.1   0:14.74 python
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.