Python C Api and correct handling of memory in C++ classes

Question

I'm wondering how to cope with the following problem. Inside my C++ class I have an auxiliary PyObject pointer.

class Foo
{
     public:
     // Should I new the dictionary here in constructor?
     Foo()
     {

     }
     // Must decrease the reference count or explicitly delete the dictionary?
     ~Foo()
     {
         Py_DECREF(myDictionary);
     }

     void sync()
     {
          myDictionary = PyDict_New();
          for (int i=0; i<myInternalData.size(); i++)
          {
                  PyObject *key =  PyInt_FromLong(i);
                  PyObject *val = PyInt_FromLong(myInternalData.at(i));
                  PyDict_SetItem(dict,key,val);
                  Py_DecRef(key);
                  Py_DecRef(val);
          }
     }

     private:
     PyObject *myDictionary;
     std::vector<int> myInternalData;
}

In my C++ code the myInternalData structure is occasionally updated or resized and I want to know how to cope with the proper memory allocation of my python dictionary.

I don't know how to deallocate the memory associated with it, or how to correctly keep it synchronized with my internal std::vector without corrupting the heap or provoking memory leaks.

Some helps with Python C API? Should I deallocate the PyDict with PyObject_Del and then reallocating it again? Someone suggesting another approach?

Are you calling sync frequently? From what I can understand this is a possible source of memory leaks, since you are allocating new objects. Also (I might be wrong here), I think that probably the appropriate way to deallocate the dictionary is to decrease the ref counter to zero (docs.python.org/2/c-api/refcounting.html). Did you try using valgrind to see where you are getting mem-leaks? — nvlass
– nvlass, Commented Jan 7, 2013 at 14:11

James Kanze · Accepted Answer · 2013-01-07 14:17:58Z

1

It's not clear to my why you're using a dictionary in Python, when you index with a contiguous set of integers, starting at 0. However: before using it, you'll have to do a PyDict_New to create the dictionary. Following that, when you resync, you should clear the dictionary before starting, using PyDict_Clear, rather than reallocate a new one.. Nothing else should be necessary. (If you reallocate a new one, as you do in your code, you should decrement the reference count on the old one first. But any code on the Python side which refers to the old one will continue to refer to the old one; PyDict_Clear is probably the better solution.)

Also, you should pay attention where temporary Python objects are involved. For the moment, nothing else is necessary because you only use Python (and thus, C) functions in the loop, and they cannot trigger a C++ exception. Change the code ever so slightly, and this may cease to be the case. As a general rule, I've found that you should wrap the PyObject* in a class whose destructor calls Py_DecRef, rather than call it explicitly, and maybe miss the call due to an exception.

answered Jan 7, 2013 at 14:17

James Kanze

155k20 gold badges191 silver badges338 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

linello Over a year ago

I moved the PyDict_New in the constructor and now in the sync method I only call PyDict_Clear(myDictionary). I've also added a Py_DECREF(myDictionary) in the destructor and now all the memory leaks are eliminated!

Collectives™ on Stack Overflow

Python C Api and correct handling of memory in C++ classes

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related