Debugging Python Extensions written in C

Question

I'm pretty familiar with writing C code and I'm comfortable in writing python code. I'm trying to learn how to write modules in C that can be called from Python-3.9.X on OSX 10.15.7. I've gotten a couple 'hello world' type of examples to work, but for complex examples I'm struggling to figure out how I would debug the C-extensions that I write.

MWE:

src/add.c

// The C function that actually does the work
static PyObject * add_c_func(PyObject *self, PyObject *args)
{
    int a=0;
    int b=0;
    int c=0;

    // This determines the number arguments used by add_c_func
    if (!PyArg_ParseTuple(args, "iii", &a, &b, &c))
    {
        return NULL;
    }

    printf("%i\n", a+b+c);

    Py_RETURN_NONE;
}

// This defines the function used by
static PyMethodDef AddMethods[] = {
    {"add_py_func", add_c_func, METH_VARARGS, "Add three numbers."},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef addpymod =
{
    PyModuleDef_HEAD_INIT,
    "addpymod",     /* name of module */
    "",          /* module documentation, may be NULL */
    -1,          /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
    AddMethods
};

PyMODINIT_FUNC PyInit_addpymod(void)
{
    return PyModule_Create(&addpymod);
}

setup.py :

from setuptools import setup, Extension
setup(
    name='addpymod',
    version='1.0',
    description='Python Package with Hello World C Extension',
    ext_modules=[
        Extension(
            'addpymod',
            sources=['src/add.c'],
            py_limited_api=True)
    ],
)

Compiling / installing (by default it uses clang):

python setup.py install

Trying to debug :

(py-ext-test) local: understand-python-c-ext $ gdb
GNU gdb (GDB) 10.1
.
.
.
(gdb) b add.c : 20
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (add.c : 20) pending.
(gdb) python
>import addpymod
>addpymod.add_py_func(10,10,10)   # Why didn't my breakpoint get hit?
>Quit
# This clearly failed, I'm not even sure where my stdout is

There are multiple levels of complexity here and I'm sure that I'm being tripped by more than one.

Question :

How do I debug my add.c using gdb (preferred) or possibly lldb since it was compiled by default with clang's -g option?

You misunderstand what the python command inside gdb means. If you want to debug python, you need tell gdb that it should debug the python executable, and then run it, not use the python command of gdb itself. — ssbssa
– ssbssa, Commented Apr 14, 2022 at 9:47
That is a fine point. I included that section to show that I had at least tried something. I'd also tried putting the commands in a file and then running it via gdb somefile.py, which did not work. It is still unclear how I can debug my C-extension. — irritable_phd_syndrome
– irritable_phd_syndrome, Commented Apr 15, 2022 at 3:32
As I said, you have to debug the python executable. Try gdb --args python somefile.py. — ssbssa
– ssbssa, Commented Apr 15, 2022 at 5:36

Alex Kaszynski · Accepted Answer · 2024-05-22 02:34:23Z

Ran into a similar issue on Linux using nanobind. I'm building using cmake and scikit-build-core, but the the key is to build using a debug build. Using scikit-build-core this can be set with:

[tool.scikit-build]
cmake.build-type = "Debug"

And within your CMakeLists.txt with:

target_compile_options(my_ext PRIVATE -g -O0)

Once you've compiled (e.g. pip install -e . --no-build-isolation), double check that your extension was built with debug symbols with:

$objdump --syms /path/to/your/extension.so

You should see a table of symbols. For example:

SYMBOL TABLE:
0000000000000000 l    df *ABS*  0000000000000000              crtstuff.c

You'll then be able to run gdb and set a breakpoint. For example:

$ gdb --args python test_ext.py
(gdb) b qual.cpp:4872
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (qual.cpp:4872) pending.
(gdb) run
Thread 1 "python" hit Breakpoint 1, ComputeWeights<float> (offset=0x1fe5830, neigh=0x20432e0, 
indices=0x1bb8ec0, points=0x1d6a7a0, n_neigh=108, n_points=27, fac=-0.75, num_threads=4) at /home/user/library-path/src/qual.cpp:4872                                          
4872      T *weights = new T[n_neigh];
(gdb)

Will Ayd · Accepted Answer · 2024-07-20 18:54:46Z

@Alex Kaszynski is correct that you are missing the proper compilation flags to enable debugging. Since you are using setuptools, you would more specifically have to write:

from setuptools import setup, Extension

compile_args = ["-g"]
setup(
    name='addpymod',
    version='1.0',
    description='Python Package with Hello World C Extension',
    ext_modules=[
        Extension(
            'addpymod',
            sources=['src/add.c'],
            py_limited_api=True,
            extra_compile_args=extra_compile_args,
        )
    ],
)

Here I've used -g, though you may opt for many different debug flags if you find anything else more suitable

For what it's worth, I've written extensively about debugging Python extensions on my blog, so you may find more valuable information there

Collectives™ on Stack Overflow

Debugging Python Extensions written in C

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related