Use PyObject_Realloc correctly

Question

https://github.com/python/cpython/blob/master/Include/objimpl.h#L83 says

PyObject_Realloc(p != NULL, 0) does not return NULL, or free the memory at p.

PyObject_Realloc doesn't free the memory.

I've noticed a memory leak in my program which uses PyObject_Realloc every now and then.

I tried to fix it using this:

PyLongObject *new = (PyLongObject *) PyObject_Realloc(obj, new_size);
if (new == NULL) {
  return PyErr_NoMemory();
}
if (new != obj) {
  PyObject_Free(obj);
}
obj = new;

But now I get the malloc error pointer being freed was not allocated

How am I meant to make sure my program doesn't leak memory when using PyObject_Malloc and PyObject_Realloc?

Have you ensured that the reference count of obj is 1? If there are other pointers to it in existance then reallocing it isn't going to be safe. — DavidW
– DavidW, Commented Mar 23, 2021 at 13:01
@DavidW checking Py_REFCNT(obj) before every reallocation shows it's always 1 — user3310334
– user3310334, Commented Mar 23, 2021 at 13:28
That's fine then. There's definitely cases where this this is reliably true (e.g. you've just allocated an object in C and haven't yet returned it to Python) — DavidW
– DavidW, Commented Mar 23, 2021 at 13:47
@DavidW since I think the memory leak comes from elsewhere, I made a new question with my full code. I wonder if you can spot anything: stackoverflow.com/q/66764918/3310334 — user3310334
– user3310334, Commented Mar 23, 2021 at 14:16

Antti Haapala · Accepted Answer · 2021-03-23 12:56:24Z

2

You need to use this in the exact same manner as realloc must be used:

PyObject *new = PyObject_Realloc(obj, new_size);

if (new == NULL) {
    // obj is still valid - so maybe you need to free it *here*
    PyObject_Free(obj);
    return PyErr_NoMemory();
}


// if a non-null pointer *was* returned, then obj was already
// freed and new points to the new memory area. Therefore
// assign it to obj unconditionally.
obj = new;

edited Mar 23, 2021 at 12:56

answered Mar 23, 2021 at 12:19

Antti Haapala

135k23 gold badges297 silver badges349 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user3310334 Over a year ago

"maybe you need to free it" doesn't instil confidence, what if it crashes harder after that?

user3310334 Over a year ago

"if a non-null pointer was returned, then obj was already freed" how do you know?

Antti Haapala Over a year ago

@minseong because that is the contract.

Antti Haapala Over a year ago

@minseong I would be 100 % sure that it needs to be freed there in a program written by me. WIth your program I'd be just 99 % sure. It is possible to write such a spaghetti program that it would crash if you call PyObject_Free(obj) there but work right otherwise. I can't know what you've written.

user3310334 Over a year ago

I guess the memory leak comes from elsewhere. If you have the time to spare, could you take a look at this question: stackoverflow.com/q/66764918/3310334

Collectives™ on Stack Overflow

Use PyObject_Realloc correctly

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related