When dealing with C extensions with Python, what is the best way to move data from a C++ array or vector into a PyObject/PyList so it can be returned to Python?
The method I am using right now seems a bit clunky. I am looping through each element and calling Py_BuildValue to append values to a PyList.
Is there something similar to memcpy?
If your vector holds homogenous numeric data, you're much better off creating an array.array to transfer the values to Python.
Unlike a list, an array internally stores its values as a contiguous array of native C values, but otherwise provides a Pythonic list-like interface. It use minimizes memory footprint and enables you to actually use a single memcpy call to efficiently transfer the data. Here is an example:
PyObject *
vec_to_array(std::vector<double>& vec)
{
static PyObject *single_array;
if (!single_array) {
PyObject *array_module = PyImport_ImportModule("array");
if (!array_module)
return NULL;
PyObject *array_type = PyObject_GetAttrString(array_module, "array");
Py_DECREF(array_module);
if (!array_type)
return NULL;
// array.array('d', [0.0])
single_array = PyObject_CallFunction(array_type, "c[d]", 'd', 0.0);
Py_DECREF(array_type);
if (!single_array)
return NULL;
}
// extra-fast way to create an empty array of count elements:
// array = single_element_array * count
PyObject *pysize = PyLong_FromSsize_t(vec.size());
if (!pysize)
return NULL;
PyObject *array = PyNumber_Multiply(single_array, pysize);
Py_DECREF(pysize);
if (!array)
return NULL;
// now, obtain the address of the array's buffer
PyObject *buffer_info = PyObject_CallMethod(array, "buffer_info", "");
if (!buffer_info) {
Py_DECREF(array);
return NULL;
}
PyObject *pyaddr = PyTuple_GetItem(buffer_info, 0);
void *addr = PyLong_AsVoidPtr(pyaddr);
// and, finally, copy the data.
if (vec.size())
memcpy(addr, &vec[0], vec.size() * sizeof(double));
return array;
}
Enhancing this with template specializations to support other primitive types is left as an exercise for the reader.
You have to construct a PyObject from each element prior to adding it to a sequence.
So you have to either add them one by one, or convert them all, then pass to a constructor from PyObject[].
I guess the 2nd way is slightly faster since it doesn't have to adjust the sequence's member variables after each addition.
If you can install Boost as a dependency, then you can take advantage of the boost::python::list class in order to make that conversion automatically. Here are the docs and here you can find an example of usage to convert a std::vector into a python::list.
