9

I would like to call my C functions within a shared library from Python scripts. Problem arrises when passing pointers, the 64bit addresses seem to be truncated to 32bit addresses within the called function. Both Python and my library are 64bit.

The example codes below demonstrate the problem. The Python script prints the address of the data being passed to the C function. Then, the address received is printed from within the called C function. Additionally, the C function proves that it is 64bit by printing the size and address of locally creating memory. If the pointer is used in any other way, the result is a segfault.

CMakeLists.txt

cmake_minimum_required (VERSION 2.6) 
add_library(plate MODULE plate.c)

plate.c

#include <stdio.h>
#include <stdlib.h>

void plate(float *in, float *out, int cnt)
{
    void *ptr = malloc(1024);
    fprintf(stderr, "passed address: %p\n", in);
    fprintf(stderr, "local pointer size: %lu\n local pointer address: %p\n", sizeof(void *), ptr);
    free(ptr);
}

test_plate.py

import numpy
import scipy
import ctypes

N = 3
x = numpy.ones(N, dtype=numpy.float32)
y = numpy.ones(N, dtype=numpy.float32)
plate = ctypes.cdll.LoadLibrary('libplate.so')

print 'passing address: %0x' % x.ctypes.data
plate.plate(x.ctypes.data, y.ctypes.data, ctypes.c_int(N))

Output from python-2.7

In [1]: run ../test_plate.py

passing address: 7f9a09b02320

passed address: 0x9b02320

local pointer size: 8

local pointer address: 0x7f9a0949a400

Improve this question

4 Answers 4

Reset to default
9

The problem is that the ctypes module doesn't check the function signature of the function you're trying to call. Instead, it bases the C types on the Python types, so the line...

plate.plate(x.ctypes.data, y.ctypes.data, ctypes.c_int(N))

...is passing the the first two params as integers. See eryksun's answer for the reason why they're being truncated to 32 bits.

To avoid the truncation, you'll need to tell ctypes that those params are actually pointers with something like...

plate.plate(ctypes.c_void_p(x.ctypes.data),
            ctypes.c_void_p(y.ctypes.data),
            ctypes.c_int(N))

...although what they're actually pointers to is another matter - they may not be pointers to float as your C code assumes.

Update

eryksun has since posted a much more complete answer for the numpy-specific example in this question, but I'll leave this here, since it might be useful in the general case of pointer truncation for programmers using something other than numpy.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

@eryksun Is a Windows long still only 32-bits on 64-bit Windows?
On 64-bit Windows a long is 32-bit and a long long is 64-bit.
@eryksun Strange, but I guess it has the virtue of being consistent.
Thanks! Your suggestion fixed the pointer address truncation problem. Although as you implied, the pointers do not point to the the intended float data.
I've discovered that the example will work of the arrays are created using dtype=numpy.float32
6

Python's PyIntObject uses a C long internally, which is 64-bit on most 64-bit platforms (excluding 64-bit Windows). However, ctypes assigns the converted result to pa->value.i, where value is a union and the i field is a 32-bit int. For the details, see ConvParam in Modules/_ctypes/callproc.c, lines 588-607 and 645-664. ctypes was developed on Windows, where a long is always 32-bit, but I don't know why this hasn't been changed to use the long field instead, i.e. pa->value.l. Probably, it's just more convenient most of the time to default to creating a C int instead of using the full range of the long.

Anyway, this means you can't simply pass a Python int to create a 64-bit pointer. You have to explicitly create a ctypes pointer. You have a number of options for this. If you're not concerned about type safety, the simplest option for a NumPy array is to use its ctypes attribute. This defines the hook _as_parameter_ that lets Python objects set how they're converted in ctypes function calls (see lines 707-719 in the previous link). In this case it creates a void *. For example, you'd call plate like this:

plate.plate(x.ctypes, y.ctypes, N)

However, this doesn't offer any type safety to prevent the function from being called with an array of the wrong type, which will result in either nonsense, bugs, or a segmentation fault. np.ctypeslib.ndpointer solves this problem. This creates a custom type that you can use in setting the argtypes and restype of a ctypes function pointer. This type can verify the array's data type, number of dimensions, shape, and flags. For example:

import numpy as np
import ctypes

c_npfloat32_1 = np.ctypeslib.ndpointer(
    dtype=np.float32, 
    ndim=1, 
    flags=['C', 'W'])

plate = ctypes.CDLL('libplate.so')

plate.plate.argtypes = [
    c_npfloat32_1,
    c_npfloat32_1,
    ctypes.c_int,
]

N = 3
x = np.ones(N, dtype=np.float32)
y = np.ones(N, dtype=np.float32)

plate.plate(x, y, N)  # the parameter is the array itself
Improve this answer

Comments

1

If you don't tell ctypes what type the parameters are, it attempts to infer it from the values that you pass to the function. And this inference will not always work as you need.

The recommended way to deal with this is to set the argtypes attribute of the function and so explicitly tell ctypes what the parameter types are.

plate.plate.argtypes = [
    ctypes.POINTER(ctypes.c_float), 
    ctypes.POINTER(ctypes.c_float), 
    ctypes.c_int
]

Then you can call the function like this:

plate.plate(x.ctypes.data, y.ctypes.data, N)
Improve this answer

3 Comments

@eryksun Personally I think that information would be worth including in an answer. I think argtypes is a better solution than the answer that is currently accepted. But clearly I don't know the details well enough, especially for numpy which is clearly a little special. Would you be able to add an answer. Then I'd delete this and upvote your accurate answer.
I have a similar looking problem, but not related to numpy. In my case all function prototypes are defined but it does not help. My 64 bits pointers are still truncated to 32 bits :-(
ok, my bad: I found out why the pointer was truncated in my case. The prototype of the function receiving the pointer was ok, but I had troubles with the another C function called from ctype allocating that pointer. The type of return of that function must be defined using restype = c_void_p or oter pointer or return type defaults as int. In my cas I did provide restype but I add a typo (a trailing "s" after restype) and it was ignored. The annoying part is that it can go unnoticed sometime as it works as long as only addresses in low memory are used by library.
1

Actually, You should set plate.argstype = [ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int], and then it will be ok to accept the address in c func from python.
I met the problem and I solved it as what I say.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.