Call multiple Python-extensions in c at the same time using threads

Question

I have a c program which uses 2 threads. Now I would like to call in both threads a Python function. When I only call one python function in one thread it does work, but when calling in both of them at the same time it gives this error:
Fatal Python error: GC object already tracked.

When looking around I found this is caused by the python-functions and that the Garbage Collector does not differentiate between both. Now I can't seem to find any examples regarding this topic. I did find some clues which could help but again no example on how to implement them. I think I need to let the GC know the difference between both in a way similar to a malloc().

I thought i could use the following function PyObject_GC_New(), but I don't know how to implement this.

Beneath my code, which works when you comment out the audioSend() in speakerPlay() and replace it with a print statement for example.

#include <stdio.h>
#include <Python.h>
#include <numpy/ndarrayobject.h>
#include <pthread.h>

typedef struct{
    int length;
    int duration;
}args;

void* micRecord(void* val);
void* calc(void* val);
void* speakerPlay(void* val);

npy_float** audioReceive(int length);
int audioSend(int length);

int main(){
  // init python imports
  Py_Initialize();
  import_array();

  int length = 2;
  int duration = 10;

  args* tArgs = &(args){.length = length, .duration = duration};

  pthread_t recordHandle;
  if( pthread_create(&recordHandle , NULL, micRecord, tArgs))exit( EXIT_FAILURE );

  pthread_t playHandle;
  if( pthread_create(&playHandle , NULL, speakerPlay, tArgs))exit( EXIT_FAILURE );

  void* result;
  if(pthread_join(recordHandle, &result)==-1)
  {
      exit(EXIT_FAILURE);
      printf("could not join\n");
  }

  if(pthread_join(playHandle, &result)==-1)
  {
      exit(EXIT_FAILURE);
      printf("could not join\n");
  }
  printf("Threads joined\n");

  Py_Finalize();
  return 0;
}

void* micRecord(void* rArgs){
  printf("in micRecord\n\n");

  args tArgs = *(args*)rArgs;
  int length = tArgs.length;
  int duration = tArgs.duration;

  npy_float** tempArray;

  for(int i = 0; i<duration; i++){
    tempArray = audioReceive(length);
  }

  printf("micRecord thread done\n");
  pthread_exit(NULL);
}

void* speakerPlay(void* pArgs){
  printf("in speakerPlay\n\n");

  args tArgs = *(args*)pArgs;
  int length = tArgs.length;
  int duration = tArgs.duration;

  for(int i = 0; i < duration; i++){
    audioSend(length);
    //printf("sending\n");
  }

  printf("Speaker thread done\n");
  pthread_exit(NULL);
}

npy_float** audioReceive(int length){
  //variables init
  PyObject *pName, *pModule, *pFunc, *pArgs;
  PyObject* pValue;
  npy_float** array;

  // import the .py file in which the to-call python function is located
  pName = PyUnicode_FromString("receiveSound");
  pModule = PyImport_Import(pName);
  Py_DECREF(pName);
  if(pModule == NULL) printf("its null\n");

  if (pModule != NULL) {
    // Get the reference to the to-call python function and checks if its callable
    pFunc = PyObject_GetAttrString(pModule, "recordSound");
    if (pFunc && PyCallable_Check(pFunc)) {
      // set arguments you want to pass along to the python function
      pArgs = PyTuple_New(1);
      PyTuple_SetItem(pArgs, 0, PyLong_FromLong(length));

      // call the python function and return the numpy array in pValue
      pValue = PyObject_CallObject(pFunc, pArgs);
      if (pValue == NULL) {
        Py_DECREF(pFunc);
        Py_DECREF(pModule);
        PyErr_Print();
        fprintf(stderr,"Call failed\n");
        return NULL;
      }

      // get the type description from the array content
      PyArray_Descr *descr;
      descr = PyArray_DescrFromType(PyArray_TYPE(pValue));

      // convert the numpy array to, a by c-compiler useable format, npy_float array
      if (PyArray_AsCArray(&pValue, (void*)&array, PyArray_DIMS(pValue), PyArray_NDIM(pValue), descr) < 0)  {
        PyErr_SetString(PyExc_TypeError, "error converting to c array");
        return NULL;
      }
      //printf("input\n%"NPY_FLOAT_FMT"\n%"NPY_FLOAT_FMT"\n%"NPY_FLOAT_FMT"\n%"NPY_FLOAT_FMT"\n", array[0], array[5], array[10], array[12]);
      Py_DECREF(pValue);
    }
    // if there was no such function in the .py file
    else {
        if (PyErr_Occurred())
            PyErr_Print();
        fprintf(stderr, "Cannot find function \n");
    }
    Py_XDECREF(pFunc);
    Py_DECREF(pModule);
  }

  // if there was no such .py file
  else {
    PyErr_Print();
    fprintf(stderr, "Failed to load \\n");
    return NULL;
  }

  return array;
}

int audioSend(int length){
  // init variables
  PyObject *pName, *pModule, *pFunc;
  PyObject *pValue;

  // import the .py file in which the to-call python function is located
  pName = PyUnicode_FromString("ss");
  pModule = PyImport_Import(pName);
  Py_DECREF(pName);
  if(pModule == NULL) printf("its null\n");

  if (pModule != NULL) {
      // Get the reference to the to-call python function and checks if its callable
      pFunc = PyObject_GetAttrString(pModule, "playSound");
      if (pFunc && PyCallable_Check(pFunc)) {

        // call the python function to play the sound by reading the array
        pValue = PyObject_CallObject(pFunc, NULL);

        // check if array is played
        if (pValue != NULL) {
            Py_DECREF(pValue);
        }

        // if the array was not correctly read
        else {
            Py_DECREF(pFunc);
            Py_DECREF(pModule);
            PyErr_Print();
            fprintf(stderr,"Call failed\n");
            return 1;
        }
      }

      // if no such function exists in the .py file
      else {
          if (PyErr_Occurred())
              PyErr_Print();
          fprintf(stderr, "Cannot find function \n");
      }
      Py_XDECREF(pFunc);
      Py_DECREF(pModule);
    }
    // if no such .py file exists
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \\n");
        return 1;
    }
  return 0;
}

Here are the two python code files, those are really basic. They are contained on my device in the files receiveSound.py and ss.py

receiveSound.py:

import numpy as np
from array import array

def recordSound(length=50):

    print ("-----------------RECORDING SOUND------------------")

    # Make the array `rec_array`
    recArray = np.array([[0,1,2,3,4], [5,6,7,8,9]], dtype=np.float)

    return recArray

ss.py

def playSound():

    print ("----------------SOUND IS PLAYED------------------")#recArray[49]
    return 1

and here is my makefile, with basicANC.c being my c-code

basicANCmake: basicANC.c
    gcc -o basicANC -I/usr/include/python3.6m basicANC.c -lpython3.6m -lpthread
run: basicANCmake
    PYTHONPATH=. ./basicANC

John Bollinger · Accepted Answer · 2020-03-17 14:03:30Z

1

Python protects its object space against concurrent access via a global lock, appropriately called the "Global Interpreter Lock". You need to respect and accommodate that in order to use the same embedded Python interpreter from multiple threads of the host C program. The documentation contains a section describing these requirements. There are ways to take fine-grained control of all of that, but, per the docs,

[t]he typical idiom for calling into Python from a C thread is:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

I suggest incorporating that pattern into your program by updating the micRecord() and speakerPlay() functions to wrap each call to audioReceive() and audioSend() between a PyGILState_Ensure() and a PyGILState_Release().

Update:

Additionally,

Depending on which version of Python you are using, you may need to call PyEval_InitThreads() before using any of the thread and GIL manipulation functions. You should do this after calling Py_Initialize(), in the same thread. It is safe to do that even on Python versions that do not actually require it.
You probably need the main thread to release the GIL before the other threads can acquire it. Calling the PyGILState_Check() function in the main thread will confirm (or refute) this. There are several ways you could release the GIL, but perhaps easiest would be to bracket the section of main() that creates and then joins child threads between a Py_BEGIN_ALLOW_THREADS and a Py_END_ALLOW_THREADS. Do read the docs for these macros for details on these and how to use them.

edited Mar 17, 2020 at 14:03

answered Mar 16, 2020 at 21:26

John Bollinger

191k11 gold badges103 silver badges207 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Vlovlo Over a year ago

I tried your approach, but when my threads get in a deadlock the moment they enter the PyGILStat_Ensure(). Even when only using it in one state (and just a print-statement in the other) it enters a deadlock. I tried avoiding it by using the following approach [codevate.com/blog/… PyThreadState_New(m_interpreterState) , PyEval_RestoreThread(m_state) , PyEval_SaveThread(). but i still ran into a deadlock with this one.

John Bollinger Over a year ago

It sounds like the main thread is holding the GIL, so that the child threads cannot acquire it. Alternatively, if your Python is older than 3.7 then the GIL probably has not been initialized in the first place. I have updated the answer to address these matters.

Vlovlo Over a year ago

Adding the Py_BEGIN_ALLOW_THREADS and a Py_END_ALLOW_THREADS did the job. Many thanks!! This saves me having to redo everything in C instead of python.

Collectives™ on Stack Overflow

Call multiple Python-extensions in c at the same time using threads

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related