Understanding memory leak with C++ extension for python

Question

I have been struggling to understand what I am doing wrong with the memory management of this this C++ function for a python module, but I don't have much experience in this regard.

Every time I run this function, the memory consumption of the python interpreter increases. The function is supposed to take in two numpy arrays and create new numpy arrays with the desired output.

extern "C" PyObject *radec_to_xyz(PyObject *self, PyObject *args) {
    // Parse the input arguments
    PyArrayObject *ra_arrobj, *dec_arrobj;
    if (!PyArg_ParseTuple(args, "O!O!", &PyArray_Type, &ra_arrobj, &PyArray_Type, &dec_arrobj)) {
        PyErr_SetString(PyExc_TypeError, "invalid arguments, expected two numpy arrays");
        return nullptr;
    }
    // skipping checks that would ensure:
    // dtype==float64, dim==1, len()>0 and equal for inputs, data or contiguous
    npy_intp size = PyArray_SIZE(ra_arrobj);

    // create the output numpy array with the same size and datatype
    PyObject *x_obj = PyArray_EMPTY(1, &size, NPY_FLOAT64, 0);
    if (!x_obj) return nullptr;
    Py_XINCREF(x_obj);

    // get pointers to the arrays
    double *ra_array = static_cast<double *>(PyArray_DATA(ra_arrobj));
    double *dec_array = static_cast<double *>(PyArray_DATA(dec_arrobj));
    double *x_array = static_cast<double *>(PyArray_DATA(reinterpret_cast<PyArrayObject*>(x_obj)));

    // compute the new coordinates
    for (npy_intp i = 0; i < size; ++i) {
        double cos_ra = cos(ra_array[i]);
        double cos_dec = cos(dec_array[i]);
        // compute final coordinates
        x_array[i] = cos_ra * cos_dec;
    }
    // return the arrays holding the new coordinates
    return Py_BuildValue("O", x_obj);
}

I suspect that I am getting the reference counts wrong and therefore, the returned numpy arrays don't get garbage collected. I tried changing the reference counts, but this did not help.

When I pass the X array as additional argument from the python interpreter instead of allocating it in the function, the memory leak is gone as expected.

Could you reduce the code some (cf. minimal reproducible example)? If you hardcode the input, you should be able to get rid of error checks, and it should be possible to drop computing coordinates without affecting memory consumption. How about dropping the "y" and "z" parts to focus on "x"? That might reduce the leak to a third, but it's still a leak, so good enough for this question. — JaMiT
– JaMiT, Commented Nov 24, 2023 at 16:35
OK, if the problem is gone in your simplified version, now incrementally add back the code until the issue returns. — nicomp
– nicomp, Commented Nov 24, 2023 at 17:12
why are you using a Py_XINCREF in there ? functions that return a new object should return it with a refcount of 1, you don't have to increment it unless you are storing it somewhere else. — Ahmed AEK
– Ahmed AEK, Commented Nov 24, 2023 at 17:13
Py_BuildValue also increments the refcount, so you should decref after you create the returned object with Py_BuildValue ... it is not needed here anyway, just return x_obj. — Ahmed AEK
– Ahmed AEK, Commented Nov 24, 2023 at 17:16

AndyG · Accepted Answer · 2023-11-24 17:24:08Z

2

Py_XINCREF(x_obj) is not needed because Py_BuildValue("O", x_obj) will increment the reference count for you. In other words, you're accidentally increasing the reference count too much by 1

answered Nov 24, 2023 at 17:24

AndyG

41.4k8 gold badges122 silver badges155 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

van Over a year ago

Thank you so much, I was not aware of this and that fixed my issue.

Collectives™ on Stack Overflow

Understanding memory leak with C++ extension for python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related