What is the most conventional way to integrate C code into a Python library using distutils?

Question

Many well-known python libraries are basically written in C (like tensorflow or numpy) because this apparently speeds things up a lot. I was able to very easily integrate a C function into python by reading this. Doing so I can finally use distutils to access the functions of the source.c file:

# setup.py

from distutils.core import setup, Extension

def main():
    setup(
        # All the other parameters...
        ext_modules=[ Extension("source", ["source.c"]) ]
        )

if __name__ == "__main__":
    main()

so that when i run python setup.py install i can install my library. However, what if i want to create a python-coded wrapper object for the functions inside source.c? Is there a way to do this without polluting the installed modules?
Wandering around the internet I have seen some seemingly simple solutions using shared libraries (.so). However I would need a solution that does not involve attaching the compiled C code, but one that compiles it the first time the program is run.

jordanvrtanoski · Accepted Answer · 2021-03-09 11:29:48Z

1

The shared libraries are the right way to go in this case. The distutils have ability to build the static libraries as follows:

from distutils.ccompiler import new_compiler
from distutils import sysconfig

c = new_compiler()
workdir = "."
c.add_include_dir( sysconfig.get_python_inc() )
c.add_include_dir("./include")
objects = c.compile(["file1.c", "file2.c"])
c.link_shared_lib(objects, "mylibrary", output_dir=workdir)

This will generate the .so library in the working directory.

For example how it's used in real setup see the following example

edited Mar 9, 2021 at 11:29

answered Mar 7, 2021 at 13:57

jordanvrtanoski

5,6271 gold badge25 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

Giuppox Over a year ago

hi thanks for answering my question. What are c.add_include_dir("./include") and c.link_shared_lib(objects, "mylibrary", output_dir=workdir) supposed to do? Also, i tried to execute your code, and it generates me two files, one .so and one .o, what's the difference?

Giuppox Over a year ago

ok thanks, what about "c.add_include_dir("./include")"?

Giuppox Over a year ago

is there a way to make the .o files be generated in a directory different from their .c files?

Giuppox Over a year ago

the project is here github.com/Giuppox/block , the thing is not that complicated, i just use setup.py to compile block/core.c (where i try to #include <Python.h>) inside a generated folder block/build. Sorry for the huge amount of questions, and thanks for your help!

Giuppox Over a year ago

hi it is also possible to do something similar using setuptools? i was reading some articles and it seems like it is the standard now...

|

Collectives™ on Stack Overflow

What is the most conventional way to integrate C code into a Python library using distutils?

1 Answer 1

12 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

12 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related