Global Interpreter Lock and access to data (eg. for NumPy arrays)

Question

I am writing a C extension for Python, which should release the Global Interpreter Lock while it operates on data. I think I have understood the mechanism of the GIL fairly well, but one question remains: Can I access data in a Python object while the thread does not own the GIL? For example, I want to read data from a (big) NumPy array in the C function while I still want to allow other threads to do other things on the other CPU cores. The C function should

release the GIL with Py_BEGIN_ALLOW_THREADS
read and work on the data without using Python functions
even write data to previously constructed NumPy arrays
reacquire the GIL with Py_END_ALLOW_THREADS

Is this safe? Of course, other threads are not supposed to change the variables which the C function uses. But maybe there is one hidden source for errors: could the Python interpreter move an object, eg. by some sort of garbage collection, while the C function works on it in a separate thread?

To illustrate the question with a minimal example, consider the (minimal but complete) code below. Compile it (on Linux) with

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -fPIC -I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/include/python2.7 -c gilexample.c -o gilexample.o
gcc -pthread -shared gilexample.o -o gilexample.so

and test it in Python with

import gilexample
gilexample.sum([1,2,3])

Is the code between Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS safe? It accesses the contents of a Python object, and I do not want to duplicate the (possibly large) array in memory.

#include <Python.h>
#include <numpy/arrayobject.h>

// The relevant function
static PyObject * sum(PyObject * const self, PyObject * const args) {
  PyObject * X;
  PyArg_ParseTuple(args, "O", &X);
  PyObject const * const X_double = PyArray_FROM_OTF(X, NPY_DOUBLE, NPY_ALIGNED);
  npy_intp const size = PyArray_SIZE(X_double);
  double * const data = (double *) PyArray_DATA(X_double);
  double sum = 0;

  Py_BEGIN_ALLOW_THREADS // IS THIS SAFE?

  npy_intp i;
  for (i=0; i<size; i++)
    sum += data[i];

  Py_END_ALLOW_THREADS

  Py_DECREF(X_double);
  return PyFloat_FromDouble(sum);
}

// Python interface code
// List the C methods that this extension provides.
static PyMethodDef gilexampleMethods[] = {
  {"sum", sum, METH_VARARGS},
  {NULL, NULL, 0, NULL}     /* Sentinel - marks the end of this structure */
};

// Tell Python about these methods.
PyMODINIT_FUNC initgilexample(void)  {
  (void) Py_InitModule("gilexample", gilexampleMethods);
  import_array();  // Must be present for NumPy.
}

I did things like this in the past, and I found that the easiest way to make this work is using ctypes to call your C functions. Give your C functions a pure C interface without any reference to Python or NumPy, and write trivial wrappers in Python that accept NumPy arrays and translate them to the appropriate C parameters. I gave an example on how to do this in this answer. — Sven Marnach
– Sven Marnach, Commented Jan 11, 2012 at 21:19
@Sven: Do you know whether ctypes makes a working copy of the array in memory? (1) Yes. In this case, I don't want it since I am dealing with large input arrays. (2) No. Then my question whether you can lift the GIL remains valid. However, in case (2), the ctypes behavior would be a hint that lifting the GIL is probably not problematic, also in code which does not use ctypes. Does anyone know whether (1) or (2) holds? — Daniel
– Daniel, Commented Jan 11, 2012 at 22:06
No, ctypes does not make a copy of the arrays. And it releases the GIL for you, so you don't have to care about it. The advantage of using ctypes is the simplicity -- you have to extract all necessary meta-information from the NumPy array while still in Python, and the GIL is released at just the right moment. I used this approach for concurrently accessing the data in NumPy arrays from multiple threads. (Note that concurrent write access to the same memory is never save.) — Sven Marnach
– Sven Marnach, Commented Jan 11, 2012 at 23:01
Thanks, Sven, this is the most helpful comment for me so far, even if it digresses a little from the original question to ctypes. If I could mark a comment as the ‘accepted answer’, this would be it. I'd like to know whether ctypes does more internally to protect the memory than what the explicit little example above does, but for a practical solution, it's sufficient for now to know that there exists a good way with ctypes. — Daniel
– Daniel, Commented Jan 12, 2012 at 0:32
I wrote this as a comment rather than an answer because it digresses. ctypes doesn't do anything to protect the memory. The code for passing the NumPy array as a parameter to a C function is actually in NumPy, not in ctypes -- ctypes is part of the standard library and isn't aware of NumPy. All the code does is extract the pointer to the underlying data and pass it on to ctypes, so ctypes doesn't even know the size of the data. I'll write an more to-the-point answer shortly. — Sven Marnach
– Sven Marnach, Commented Jan 12, 2012 at 14:00

Fred Foo · Accepted Answer · 2012-01-11 19:52:44Z

6

Is this safe?

Strictly, no. I think you should move the calls to PyArray_SIZE and PyArray_DATA outside the GIL-less block; if you do that, you'll be operating on C data only. You might also want to increment the reference count on the object before going into the GIL-less block and decrement it afterwards.

After your edits, it should be safe. Don't forget to decrement the reference count afterwards.

edited Jan 11, 2012 at 19:52

answered Jan 11, 2012 at 18:52

Fred Foo

365k80 gold badges765 silver badges852 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Daniel Over a year ago

Thanks for the comment about PyArray_SIZE and PyArray_DATA. That was a mistake. I edited my question and moved the commands outside the block where the GIL is released.

Daniel Over a year ago

Now: is the code in its revised version safe? Can you elaborate why incrementing the reference count would change things?

Fred Foo Over a year ago

@Daniel: the reference count might be needed (though I'm not entirely sure in this case) because no other thread must deallocate the array.

Daniel Over a year ago

I checked the reference counts: PyArray_FROM_OTF increases the reference count already, so there is no need to do it manually. However, this brings to my attention that a Py_DECREF is in order at the end of my example code; otherwise there would be a memory leak. Thanks, larsmans! I corrected my original posting. So, as an answer to your original comment: if I incremented the reference counter manually, this would be unnecessary and would not change anything, since the reference counter is at least 1 and thus the object is safe from deallocation.

David Heffernan · Accepted Answer · 2012-01-11 18:40:44Z

4

Can I access data in a Python object while the thread does not own the GIL?

No you cannot.

answered Jan 11, 2012 at 18:40

David Heffernan

616k46 gold badges1.1k silver badges1.5k bronze badges

1 Comment

Daniel Over a year ago

Does that mean that I always have to copy the contents of a NumPy array, even if I only want to read the data in a thread? I hope that there is a way around it! Any suggestion?

Collectives™ on Stack Overflow

Global Interpreter Lock and access to data (eg. for NumPy arrays)

2 Answers 2

4 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related