20

I have a C++ library that has a Python wrapper (written with SWIG). This library allows executing small user-defined code (a callback), such as element-wise operations on a vector. I.e. instead of just a + you can do whatever arbitrary binary function. Right now this is accomplished by accepting a callable Python object for the binary function and calling it. It works, but is about 80 times slower than code that doesn't have to bounce up and down into Python at every iteration.

How would I write/build/import a Cython function could be passed into my C++ library so that it can be called directly by the C++ library?

Edit: If I just stuck to C then I would write something like 

EWise(double (*callback)(double, double))

EWise would then callback(10, 20); or such. I want callback to be written in Cython, using whatever name the user wants, and a pointer to it has to be passed to my C++ library through Python somehow. That somehow is where I'm unclear.

Improve this question

3 Answers 3

Reset to default
18

The trick with cython is in using the keyword public

cdef public double cython_function( double value, double value2 ):
    return value + value2

Then the command cythonize <your_file.pyx> along with <your_file.c> will create header <your_file.h> that you can include. Alternatively, you can create the header yourself:

#ifdef __cplusplus {
extern "C"
#endif

double cython_function( double value, double value2 );

#ifdef __cplusplus
}
#endif

Update:

Then with a little overlay from Python you can use ctypes's callback mechanism

func_type = CFUNCTYPE(c_double, c_double, c_double)

your_library.set_callback_function ( func_type(user_modules.cython_function) )
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

That's what I want to accomplish, but since cython_function has to be user defined (not written by me), I won't know its name ahead of time. I need to support multiple such functions so I can't force a name either.
Do you know about numexpr ? It compiles python functions code.google.com/p/numexpr
That looks interesting, I'll have to experiment with that. But my hands are tied for this project because our vectors are written in C++. It's a distributed memory system and it's got too many pieces to rip out its vectors and replace them with Python vectors :(
If you dont know the name of an object, how are you gonna be able to call it ? Post an example of what you want to achieve that explains a bit better the constraints. I think you would have to compile it at runtime, parse the source and get the name of the functions defined.
2

You can achieve that by doing pure cdef functions :

# declare the prototype of your function
ctypedef void (*callback_ptr)(int arg)

# declare your function as cdef
cdef void my_callback(int arg):
    print 'doing some python here', arg

# now, you can use the cdef func as a callback
# in pure C !
cdef void run():
    cdef callback_ptr p = my_callback
    p(42)

if __name__ == '__main__':
    run()

Note: you can use "cython -a" to see that they are no python code involved for the content of run. So it will work with your c library.

Improve this answer

8 Comments

But I don't know that the callback is called my_callback. Remember that I don't write my_callback, my users do, and they need to be able to write any number of callback functions.
If your user write the callback in python, you have no way for doing that, except converting their python code to sort of C code. Using Cython, or any other parser tool.
I'm interested in the case where my user writes a callback in Cython. They compile it. Using whatever Python or C++ trickery I want a pointer to their Cython function.
Ok, so if you don't even know the prototype of the user func, a func is still a void*. Use their Cython file directly, and do <long>&func (for eg.)
What do you mean use their Cython file directly?
|
0

Embedding Python in Another Application may be useful reading.

Improve this answer

4 Comments

That means going through PyObject_CallObject() and friends to call the function, which is something I want to avoid. I want to call the C function directly.
Your biggest issue is you don't know how to do PyObject -> C right ? Cython + early declaration can help you no ? Declare your function in cython starting with cdef. You'll have a c definition, and you can still use Python in the C function.
I can do PyObject -> C just fine. But that's slow. The point is to avoid building PyTuples and using PyObject_CallObject(). That's where the bulk of the overhead appears to be.
@tito, right I can write a C function in Python syntax with a cdef. But how do I get a pointer to that function so that my C++ code can call it directly, thus avoiding building argument tuples and other Python overhead?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.