Data transfer between C++ and Python

Question

I would like to share memory between C++ and Python.

My problem:

I am working with big data sets (up to 6 GB of RAM) in C++. All calculations are done in c++.
Then, I want to "paste" all my results to a Python program. But I can only write my data on disk, and then read that file from Python, which is not efficient.

Is there any way to "map" memory corresponding to C++ variables so that I may access the data from Python? I don't want to copy 6GB of data onto a hard drive.

You may want to search Google/Bing for "InterProcess Communication" (IPC); and you may want to see this Stackoverflow post — WhiZTiM
– WhiZTiM, Commented Oct 16, 2016 at 12:29
Depending on how large a code base you have in C++, it might be an idea to use something like Cython. — Bendik
– Bendik, Commented Oct 16, 2016 at 12:41
@WhiZTiM thx, currently i am thinking about IPC method: Shared memory, leater about sockets or mapping file to memory. I know that python dont have pointers, but is there any way to access memory via address ? — PatrykB
– PatrykB, Commented Oct 16, 2016 at 12:41
@Bendik about 7k of c++ lines, and it is still growing. atm i am thinking about reading data in python via memory adress — PatrykB
– PatrykB, Commented Oct 16, 2016 at 12:43
You might want to embed the python interpreter in your c++ program, that would allow you to give python code access to its data using python APIs. — spectras
– spectras, Commented Oct 16, 2016 at 13:02

Community · Accepted Answer · 2020-06-20 09:12:55Z

4

First path: I think the more appropriate way for you to go is ctypes. You can create a shared library, and then load the functions of the shared library in Python, and fill all the data containers you want in Python.

In Windows, you can create a DLL, and in Linux you can create a shared .so library.

Now this has the advantage that this will be independent of your Python version.

Second path: I think it's less appropriate but you can get it to work, which is the Python C Extension. With this, you can call Python data containers (PyObjects) and fill them inside C.

However, the code you compile here will always need to be linked to Python libraries.

Which one to use?:

Use ctypes if you have some functions you want to call in C/C++, and then do the rest of the work in Python.
Use Python C Extension if you have some functions you want to call in Python, and you want to do the rest in C/C++.

With both options, you can transfer huge blocks of memory between C++ and Python without necessarily involving any disk read/write operations.

Hope this helps.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Oct 16, 2016 at 13:26

The Quantum Physicist

26.6k20 gold badges125 silver badges214 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

PatrykB Over a year ago

Yup it helps. Thank you, btw great nick.

The Quantum Physicist Over a year ago

@cukier9a7b5 Glad it helps. Btw, there are many wrappers that make using C Extension easier, like Boost and SWIG. Consider these too. I just remember them now. Cheers!

Collectives™ on Stack Overflow

Data transfer between C++ and Python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related