how to pass float* array to C method from python script whose memory allocation takes place in C layer

Question

I am trying to call C methods from Python script, C method calls inturn the C++ method. I am allocating array inside the getResults() method using malloc(). Now the issue is how to pass the arguments to float* oresults in python script whose memory allocation takes place inside the C layer. This is io.c

int getResults(char* iFilename, char* iStagename, int iStateidCnt, 
    int* Stateids, int iEntityIdCount, int* iEntityids, char* iEntityType,
    char* iVariablegroup, char* ivariable, int *oRescount,
    float* oResults)
{
    int Status, i;
        EString etype(iEntityType), stagename(iStagename);
    EString vargroup(iVariablegroup);
    std::vector<ERF_INT> entity_ids;
    std::vector<ERF_INT> stateids;
    std::vector<ERF_FLOAT> results;
    _CopyIntArrayIntoVector(iStateidCnt, Stateids, stateids);
    _CopyIntArrayIntoVector(iEntityIdCount, iEntityids, entity_ids);
    CreateIoInstance(iFilename, iStagename);
    ioData pIodata = CreateIoDataInstance();
    if (iEntityIdCount <= 0)
        pIodata.setWholeSection(true);
    else
    {
        pIodata.setWholeSection(false);
        pIodata.setEntityList(entity_ids);
    }
        
    pIodata.setStateList(stateids);
    pIodata.setType(etype);
    pIodata.setVariableGroup(iVariablegroup);
    pIodata.setVariable(ivariable);
        //This is C++ method
    Status = pIo->get_results(pIodata, results);
    *oRescount = results.size();
        //allocation for oresults whose size > 2
    oResults = (float*)malloc(results.size() * sizeof(float));
    _CopyVectorIntoDoubleArray(results, oResults);
    return Status;
}

test.py

from ctypes import *
import os, sys
dll = CDLL('D:\\erf_utils_python\\erf_utils_io.dll')
dll.getresults.argtypes = (c_char_p,c_char_p,c_int,POINTER(c_int),c_int,POINTER(c_int),c_char_p,
                                  c_char_p,c_char_p,POINTER(c_int),POINTER(c_float))
dll.getresults.restype = c_int


def make_array(ctype,arr):
    return len(arr),(ctype * len(arr))(*arr)

def getresults(filename,stagename,sids,eids,entitytype,groups,variables):
    if(len(sids)>0):
       stateidcount,stateids = make_array(c_int,sids)
    if(len(eids)>0):
       entityidcount,entityid = make_array(c_int,eids)
    oresultlen = c_int()
    float_values = POINTER(c_float)
    err = dll.getresults(filename,stagename,stateidcount,stateids,entityidcount,entityid,
                                entitytype,groups,variables,byref(oresultlen), byref(float_values))
    return err,oresultlen.value, float_values

filename = b'D:\\inputfile.h5'
stagename = b"post"
stateids = [2]
stateidcount = 1
entityidcount = 1
entityid = [1]
entitytype = b"test"
variablecount = 1
variablegroup = b"testdata"
variable = b"next"

err,oreslen,ores = getresults(filename,stagename,stateids,entityid,entitytype,variablegroup,variable)

TypeError: byref() argument must be a ctypes instance, not '_ctypes.PyCPointerType' this is the error I get when I run the script. I am little confused on how to send argument for float *oresults in script.

Pascal Getreuer · Accepted Answer · 2020-10-12 06:37:23Z

1

In the C++ code, the signature int getResults(..., float* oResults) isn't able to communicate the allocated pointer back to the caller. The line

oResults = (float*)malloc(results.size() * sizeof(float));

sets the oResults pointer locally within getResults, without affecting the caller. In order to output a pointer, you must either return it or use a pointer-to-pointer argument: int getResults(..., float** oResults).

In the Python code, I'm not familiar with ctypes but it looks like float_values = POINTER(c_float) is a problem. POINTER(c_float) creates a Python type for a pointer to float. You'd want POINTER(c_float)() to make an instance of such a pointer (which is initially null).

ctypes documentation on pointers: https://docs.python.org/3/library/ctypes.html#pointers

answered Oct 12, 2020 at 6:37

Pascal Getreuer

3,2891 gold badge7 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mark Tolonen · Accepted Answer · 2020-10-12 07:11:34Z

0

The float* oResults parameter is passed by value, so it is impossible to return an allocated pointer in that parameter. Instead, use a float** oResults.

Also, float_values = POINTER(c_float) is a type, not an instance of a type. So byref(float_values) is equivalent to the invalid C &(float*). Instead, you want an instance of a pointer POINTER(c_float)() (note parentheses) and pass that by reference, similar to C float *p; func(&p). This will pass the pointer by address to the C function, which can then modify it as an output parameter.

Here's a simplified example focusing on just the int *oRescount and float** oResults parameters. Also needed is a function to free the allocation:

test.cpp

#include <vector>
#define API __declspec(dllexport)

extern "C" {
    API int getResults(size_t *oRescount, float** oResults) {
        std::vector<float> results {1.25,2.5,3.75,5.0}; // Simulated results
        *oRescount = results.size(); // Return size of results
        auto tmp = new float[results.size()]; // allocate
        for(size_t i = 0; i < results.size(); ++i) // copy vector to allocation
            tmp[i] = results[i];
        *oResults = tmp; // return allocation
        return 0;
    }

    API void freeResults(float* oResults) {
        delete [] oResults;
    }
}

test.py

from ctypes import *
dll = CDLL('./test')
dll.getResults.argtypes = POINTER(c_size_t),POINTER(POINTER(c_float))
dll.getResults.restype = c_int

def getresults():
    oRescount = c_size_t()         # instance to hold the returned size
    oResults = POINTER(c_float)()  # instance of a float* to hold the returned allocation.
    err = dll.getResults(byref(oRescount), byref(oResults))

    # oResults is a float* and it is possible to index past the end.
    # Make a copy into a Python list slicing to the correct size,
    # then free it so there is no memory leak.
    results = oResults[:oRescount.value]
    dll.freeResults(oResults)

    return err,results

err,ores = getresults()
print(err,ores)

Output:

0 [1.25, 2.5, 3.75, 5.0]

edited Oct 12, 2020 at 7:11

answered Oct 12, 2020 at 6:48

Mark Tolonen

181k26 gold badges182 silver badges278 bronze badges

3 Comments

sand Over a year ago

I have allocated memory as you have shown here and I get OSError: exception: access violation reading 0x0000000000000000..in python...when I tested only C method the line *oResults = temp gives nullptr exception.is there any other way to allocate memory in C like using malloc

sand Over a year ago

I tried your implementation for getresults() and it works well, but when I use my implementation it throws error , I guess std::vector<float> results {1.25,2.5,3.75,5.0}; this is static allocation and it works but in my situation I am getting vector results from another C++ method.is it this is the problem, please do help me

Mark Tolonen Over a year ago

@sand The implementation is generic And the vector is a local variable. The result is copied to the heap-allocated memory. Generate random numbers of a random size and it will still work.

Collectives™ on Stack Overflow

how to pass float* array to C method from python script whose memory allocation takes place in C layer

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related