C function name-dependent segfault with Python ctypes

Question

I'm getting a really weird crash when using ctypes in Python, but I'm not sure if the problem comes from Python or C.

Here is the C source (in test.c):

#include <stdio.h>

void compress(char *a, int b) {
  printf("inside\n");
}

void run() {
  printf("before\n");
  compress("hi", 2);
  printf("after\n");
}

Then here's what happens when I call run() with ctypes:

$ python -c 'import ctypes; ctypes.cdll.LoadLibrary("./test.so").run()'
before
Segmentation fault (core dumped)

The weirdest thing is that the crash doesn't happen when I rename compress() to anything else.

Other things that prevent it from crashing:

Calling compress() directly
Calling run() or compress() from C directly (If I add a main(), compile it directly, and execute it)
Removing either argument from the signature of compress() (but then the function doesn't seem to execute, based on the lack of "inside" being printed.

I'm pretty new to C, so I'm assuming there's something I'm missing here. What could be causing this?

System info:
Python 2.7.6
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
Ubuntu 14.04
uname -r: 3.13.0-58-generic

This sounds like there's a different compress function that ends up being called. — melpomene
– melpomene, Commented Jul 29, 2015 at 0:59

falsetru · Accepted Answer · 2015-07-29 01:01:45Z

6

According to the debugging, the program is trying to call compress in libz.so.1.

$ gdb python -c core
...
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python -c import ctypes; ctypes.cdll.LoadLibrary("./test.so").run()'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f9ddea18bff in compress2 () from /lib/x86_64-linux-gnu/libz.so.1

which accepts different parameters (zlib.h):

ZEXTERN int ZEXPORT compress OF((Bytef *dest,   uLongf *destLen,
                                 const Bytef *source, uLong sourceLen));

ZEXTERN int ZEXPORT compress2 OF((Bytef *dest,   uLongf *destLen,
                                  const Bytef *source, uLong sourceLen,
                                  int level));
/*

You can modify the compress function to be static to work around the issue:

static void compress(char *a, int b)
{
    printf("inside\n");
}

answered Jul 29, 2015 at 1:01

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Eryk Sun Over a year ago

If compress should be exported, you can use a gcc function attribute to mark the symbol visibility as protected, e.g. void __attribute__ ((visibility("protected"))) compress(char *a, int b). Via readelf -s test.so you'll see the symbol visibility as PROTECTED. This way you can still call compress via ctypes, which you can't do if you make it static.

Nick S Over a year ago

Thank you, I suspected that somehow it was colliding with another function named compress. But I'm still not clear on how. Does it involve libraries the Python interpreter uses? How is a function from there getting into the namespace of my ctypes-imported library?

Eryk Sun Over a year ago

@NickS, C doesn't let you control symbol visibility other than making a symbol static to a file. Thus your compress function has "default" visibility, which allows it to be overridden. Using "hidden" is preferred for non-API functions since it forces the binding to local instead of global, which is more efficient. Using "default" visibility is preferred for exported APIs; just use a unique prefix. "protected" is the most expensive implementation (because C function pointers). Read "How To Write Shared Libraries",Ulrich Drepper, 2011.

Patrick Maupin · Accepted Answer · 2015-07-29 01:52:00Z

While @falsetru has diagnosed the problem, his solution won't work in the general case where you have a lot of files to statically link together (because the entire point of declaring things static is to not have them visible from other files).

And while @eryksun has posted a solution for when you want to declare a function the same name as another, in general, you may have a lot of C functions you don't want to export, and you don't want to have to worry about whether they collide with some random function in some library that Python happens to import, and you don't want to have to prefix every one of your internal functions with an attribute.

(GCC maintains documentation on function attributes, including this function visibility feature.)

A more general solution to avoiding namespace collisions is to tell the linker not to export any symbols by default, and then to mark only those functions you want exported, like run(), as visible.

There is probably a standard way to define the macro for this, but my C is so out-of-date I wouldn't know it. In any case, this will work:

#include <stdio.h>

#define EXPORT __attribute__((visibility("protected")))

void compress(char *a, int b) {
  printf("inside\n");
}

EXPORT void run() {
  printf("before\n");
  compress("hi", 2);
  printf("after\n");
}

You can link and run it like this:

$ gcc -x c test.c --shared -fvisibility=hidden -o test.so
$ python -c 'import ctypes; ctypes.cdll.LoadLibrary("./test.so").run()'
before
inside
after

This is great. I wish there was an answer I could accept that incorporated the work of all 3 of you (you, @falsetru, and @eryksun)!

Collectives™ on Stack Overflow

C function name-dependent segfault with Python ctypes

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related