C-extension in Python - pyObject called Py_DECREF ,reference is 0,but memory leak

Question

this is my code.

PyObject *dataPyParams = PyList_New(0);
for (int i = 0; i < figdata.dataSetList.size(); i++)
{
    PyObject *pyParams = PyList_New(0);
    for (int j = 0; j < figdata.dataSetList[i].size(); j++)
    {
        //std::cerr << figdata.dataSetList[i][j] << "data\n";
        auto temp = Py_BuildValue("f", figdata.dataSetList[i][j]);
        PyList_Append(pyParams,temp);
        Py_DECREF(temp);    
    }
    PyList_Append(dataPyParams, pyParams);
    Py_DECREF(pyParams);
    
}
Py_DECREF(dataPyParams);

i called Py_DECREF(dataPyParams),dataPyParams reference is 0 ,but memory was not free. I try delete PyList_Append(pyParams,temp),this free up memory.It bothers me a lot.

How do you measure memory? How do you check that the "memory was not free"? — Some programmer dude
– Some programmer dude, Commented Nov 18, 2020 at 3:05
I think you can replace the inner loop with PyList_SetItem(pyParams, Py_BuildValue(...)) because it will take the reference. — Aaron D. Marasco
– Aaron D. Marasco, Commented Nov 18, 2020 at 3:05
That's not reliable or good enough to measure actual memory leaks. Try to find some memory profiling tools to help you measure actual allocations and free's. — Some programmer dude
– Some programmer dude, Commented Nov 18, 2020 at 3:20
@AaronD.Marasco: That only works if the list is presized, and you're filling in the existing, but unpopulated, indices. They used PyList_New(0), which precludes that. — ShadowRanger
– ShadowRanger, Commented Nov 18, 2020 at 3:28

ShadowRanger · Accepted Answer · 2020-11-18 03:51:24Z

2

Python (and most languages) don't directly satisfy memory allocations from the OS, nor return allocations immediately on release. It allocates blocks of memory in bulk, and satisfies smaller requests by partitioning up the blocks. Even when all memory in a block is "freed", it's not always returned to the OS, but held in reserve against future allocations.

In this case, you're making float and list objects, both of which have "free lists" of their own, so freeing doesn't actually return them to the allocator, but to a simple stack of allocated, but unused, objects that the float and list constructors can pull from more cheaply than asking the allocator for more memory. Problem is, this also means the blocks those elements are on can't be returned to the OS at all, because parts of them are still allocated, at least from the allocator's point of view. You can clear those free lists by explicitly calling PyGC_Collect() (documented only by side-effect in a What's New doc), which might allow memory to be returned to the OS, but again, it's no guarantee. You'd probably also want to disable pymalloc (to avoid additional small object arenas that are even less likely to be handed back to the OS). Even with all that though, Python would be perfectly within its rights to hold onto much of the memory indefinitely to satisfy future allocation.

In short, this probably isn't a memory leak. You can use more advanced memory profiling tools (if nothing else, Python itself will tell you about arena usage if you define PYTHONMALLOCSTATS=1 in your environment prior to launch), but the Task Manager can only see what the OS sees, not the internal memory management that Python itself layers on top of the raw, bulk OS memory allocation.

A simple test to see if this leaks with no external tooling would be to run this exact code (including the cleanup) multiple times. If the memory keeps going up by a large fixed amount each time it's run, then yeah, it's probably a leak. But more likely, you'll find that the first run consumes a significant amount of memory, but subsequent runs add little or no memory usage (some might get used due to allocation ordering and allocation alignment issues, but it would be pretty small), as they're drawing on the memory freed (in userland, not to the OS) rather than asking the OS for more memory.

edited Nov 18, 2020 at 3:51

answered Nov 18, 2020 at 3:46

ShadowRanger

158k12 gold badges221 silver badges315 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

KeyboardMan Over a year ago

A new problem has arisen that calls to PyGC_Collect() take more than ten seconds.python2.7

ShadowRanger Over a year ago

@KeyboardMan: Python 2.7? WHYYYYY?!? It's been end of life and completely unsupported for nearly a year now. Why are you writing new code for it, let alone new extension code (that's significantly less portable than Python level code)? 10 seconds does seem unusually long, but there's no way to tell what you've done to cause it. Regardless, normal code should not be calling PyGC_Collect() under normal circumstances; it'll be run on its own in stages intermittently, and forcing it is almost never needed, I just mentioned it as a way to forcibly clear free lists (a side-effect of main purpose).

Collectives™ on Stack Overflow

C-extension in Python - pyObject called Py_DECREF ,reference is 0,but memory leak

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related