Python Garbage Collection: Memory no longer needed not released to OS?

Question

I have written an application with flask and uses celery for a long running task. While load testing I noticed that the celery tasks are not releasing memory even after completing the task. So I googled and found this group discussion..

https://groups.google.com/forum/#!topic/celery-users/jVc3I3kPtlw

In that discussion it says, thats how python works.

Also the article at https://hbfs.wordpress.com/2013/01/08/python-memory-management-part-ii/ says

"But from the OS’s perspective, your program’s size is the total (maximum) memory allocated to Python. Since Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase."

And I use Linux. So I wrote the below script to verify it.

import gc
def memory_usage_psutil():
    # return the memory usage in MB
    import resource
    print 'Memory usage: %s (MB)' % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000.0)

def fileopen(fname):
    memory_usage_psutil()# 10 MB
    f = open(fname)
    memory_usage_psutil()# 10 MB
    content = f.read()
    memory_usage_psutil()# 14 MB

def fun(fname):
    memory_usage_psutil() # 10 MB
    fileopen(fname)
    gc.collect()
    memory_usage_psutil() # 14 MB

import sys
from time import sleep
if __name__ == '__main__':
    fun(sys.argv[1])
    for _ in range(60):
        gc.collect()
        memory_usage_psutil()#14 MB ...
        sleep(1)

The input was a 4MB file. Even after returning from the 'fileopen' function the 4MB memory was not released. I checked htop output while the loop was running, the resident memory stays at 14MB. So unless the process is stopped the memory stays with it.

So if the celery worker is not killed after its task is finished it is going to keep the memory for itself. I know I can use max_tasks_per_child config value to kill the process and spawn a new one. Is there any other way to return the memory to OS from a python process?.

That article is BS actually. The memory is likewise returned to the OS in Linux. Earlier this week there was this question about a program where slowdown was caused by Python returning memory to OS too eagerly — Antti Haapala
– Antti Haapala, Commented Apr 15, 2016 at 11:56
However the problem is that the heuristics of the arena allocator usually are bad by itself and it does not manage to free full arenas then. — Antti Haapala
– Antti Haapala, Commented Apr 15, 2016 at 11:58

Community · Accepted Answer · 2017-04-13 12:36:24Z

3

I think your measurement method and interpretation is a bit off. You are using ru_maxrss of resource.getrusage, which is the "high watermark" of the process. See this discussion for details on what that means. In short, it is the peak RAM usage of your process, but not necessarily current. Parts of the process could be swapped out etc.

It also can mean that the process has freed that 4MiB, but the OS has not reclaimed the memory, because it's faster for the process to allocate new 4MiB if it has the memory mapped already. To make it even more complicated programs can and do use "free lists", lists of blocks of memory that are not in active use, but are not freed. This is also a common trick to make future allocations faster.

I wrote a short script to demonstrate the difference between virtual memory usage and max RSS:

import numpy as np
import psutil
import resource


def print_mem():
    print("----------")
    print("ru_maxrss: {:.2f}MiB".format(
            resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024))
    print("virtual_memory.used: {:.2f}MiB".format(
            psutil.virtual_memory().used / 1024 ** 2))


print_mem()
print("allocating large array (80e6,)...")
a = np.random.random(int(80e6))

print_mem()
print("del a")
del a

print_mem()
print("read testdata.bin (~400MiB)")
with open('testdata.bin', 'rb') as f:
    data = f.read()

print_mem()
print("del data")
del data

print_mem()

The results are:

----------
ru_maxrss: 22.89MiB
virtual_memory.used: 8125.66MiB
allocating large array (80e6,)...
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8731.85MiB
del a
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8121.66MiB
read testdata.bin (~400MiB)
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8513.11MiB
del data
----------
ru_maxrss: 633.20MiB
virtual_memory.used: 8123.22MiB

It is clear how the ru_maxrss remembers the maximum RSS, but the current usage has dropped in the end.

Note on psutil.virtual_memory().used:

used: memory used, calculated differently depending on the platform and designed for informational purposes only.

edited Apr 13, 2017 at 12:36

CommunityBot

11 silver badge

answered Apr 15, 2016 at 7:37

Ilja Everilä

53.4k9 gold badges138 silver badges142 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Shinto C V Over a year ago

But htop, top and 'cat /proc/<pid>/stat' is showing the same 14 MB RES. Are htop and top also showing the max RES as well and not the current usage? Also should not we be worried about the RAM usage rather than the virtual memory used?

Ilja Everilä Over a year ago

In that our systems differ. Using your script with an added raw_input() right after content = f.read() to stop progression for observing RSS I see it raise to 10MiB for the time of the function fileopen, after which it drops back to 6.7MiB. Virtual memory usage can be a problem for 32bit systems, if it accumulates per process.

Ilja Everilä Over a year ago

Also an interesting question is: does the RSS raise, if you repeat fun(sys.argv[1]) in a (infinite) loop? Again on this machine the answer is no, it's a stable 10.8MiB, but this time when I break the loop, the memory is not reclaimed by the OS and I see the same effect of RSS staying at 10.8MiB. Perhaps the OS decided that since it seems the program constantly needs that memory, why reclaim it in between.

Ilja Everilä Over a year ago

It seems that even calling fun(...) twice in a row is enough to trigger the effect.

Shinto C V Over a year ago

Which OS do you use? On Windows it does return to OS. On linux it does not. I tried it calling twice and yes the memory does not grow. It stays there. I would prefer if it reduce after the use. But that is not happening.

|

Collectives™ on Stack Overflow

Python Garbage Collection: Memory no longer needed not released to OS?

1 Answer 1

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related