C++ extension to Python - safe memory access, and memory layout

Question

I'm extending python code with c++ functions acting on Numpy arrays (very large). Due to legacy i currently have both PyBind and Python API functions, both for Python 3.6 and above. As soon as I access memory via ptr, I would love to be sure that memory layout is exactly corresponding to c++ array under this ptr. I found that transposed array has exactly the same content in ptr in both cases. I also found that subarrays being send via Python API give in c++ exactly the same ptr as if it was the full array. In the course of development and testing i also observed more weird examples I believe, but cannot reproduce them any longer. I cannot find any recipes on the internet so far. My solution is to make copy of all input arrays in Python, like f(a.copy(), b.copy())

It seem to work well. Is this optimal/sufficient solution? I do not have any limitations on how input arrays have been produced. Transpose, subarray, reshape, in any combinations.

Have you checked out using the buffer protocol with Pybind? pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html — Matt Eding
– Matt Eding, Commented Jan 7, 2020 at 16:42
What does “memory layout is exactly corresponding to c++ array” mean? What situation are you trying to avoid? — Davis Herring
– Davis Herring, Commented Jan 8, 2020 at 4:21
PyBind11 code is using NDArray<double> xja(xj); xja.ptr As far as I understand, it is what Matt essentially means by the link template <typename T> struct NDArray { NDArray(py::array_t<T> &X) { auto buf=X.request(); ndim=buf.ndim; size=buf.size; for (int d=0; d<ndim; d++) { shape.push_back(buf.shape[d]); } ptr=(T*)buf.ptr; } @Davis: I need to be sure that a = np.array([[1.,2.],[3.,4.]]) gives pointer in c++ which reads a.ptr,4 = 1.,2.,3.,4. I can tell where it is not the case: call c++ extension function f(a=b.T). inside c++ ptr=1,2,3,4, not 1,3,2,4 — Vladimir
– Vladimir, Commented Jan 8, 2020 at 10:25
Sorry, formatting is terrible, but all my newlines seem to disappear. — Vladimir
– Vladimir, Commented Jan 8, 2020 at 10:27
@Vladimir: Often the correct way to respond to a comment on your post is to edit the post to incorporate the new information. — Davis Herring
– Davis Herring, Commented Jan 8, 2020 at 14:03

Davis Herring · Accepted Answer · 2020-01-08 14:39:06Z

2

With pybind11, you can use the py::array::c_style flag as described in Matt Eding’s link. Numpy’s C API provides much the same functionality via the NPY_ARRAY_C_CONTIGUOUS flag. In either case, the array will be copied implicitly if needed to satisfy the layout requirements; if you prefer to reject such arguments (to avoid silent inefficiency), you’ll have to check the array’s flags yourself.

answered Jan 8, 2020 at 14:39

Davis Herring

42k4 gold badges58 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Vladimir Over a year ago

This is brilliant! Thank you, exactly what I was looking for. I was blinded by the fact that default np array is c contagious in python.

Vladimir Over a year ago

Maybe one related question then? I believe I did not miss in the link page ;) Does all this in principle mean that returning values from c++ into python by modifying inputted return variable is dangerous as such? It's a pity if I cannot accumulate results of MC-like blocks in the same place and need to allocate at every block...

Davis Herring Over a year ago

@Vladimir: Many Python types are supposed to be immutable; for ndarray, writability is just another flag that might compel a copy.

Collectives™ on Stack Overflow

C++ extension to Python - safe memory access, and memory layout

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related