Why GIL is not synchrionizing Python threads that are running native C++ code inside a DLL?

Question

I have developed a simple multi-threaded Python application (Python 3.7) to call 8 different compute-intensive tasks within 8 threads . The tasks can be either a Python code or a C++ code that is embedded inside a DLL and is accessible through ctypes package. I am running the experiments on a 8-core machine on Windows.

The strange point is that when all the threads call the Python code, it seems that only one thread is active at an specific time and the CPU utilization is around 12.5%. But when calling the C++ code inside the DLL, the whole cores are used and the CPU utilization is 100%.

Now, the question is that why GIL (Global Interpreter Lock) is not synchronizing Python threads that call native C++ codes? Is the ctypes implementation releases the GIL when calling the native C++ code?

Edit 1: No macro like Py_BEGIN_ALLOW_THREADS is used inside the Native C++ DLL.

ctypes has several references to 'releasing the GIL' docs.python.org/3/library/ctypes.html — Max
– Max, Commented Dec 19, 2018 at 17:19

CristiFati · Accepted Answer · 2018-12-20 09:06:36Z

2

From [Python 3]: ctypes - Loading shared libraries (emphasis is mine; thanks @user2357112 for pointing out this very explicit quote (waay better than what I've originally posted)):

The Python global interpreter lock is released before calling any function exported by these libraries, and reacquired afterwards.

You can also find this statement in other forms on the same page (check PyDLL, CFUNCTYPE).

There are ways of going around the GIL limitation:

Replacing the threading module usage by multiprocessing ([Python 3]: multiprocessing - Process-based parallelism). This is the most common one
Enclosing code blocks that can be executed in parallel in Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS. The drawback would be that the .dll(s) will now depend on Python

edited Dec 20, 2018 at 9:06

answered Dec 19, 2018 at 18:17

CristiFati

41.6k9 gold badges68 silver badges115 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

user2357112 Over a year ago

There's a more explicit statement below the docs for CDLL, OleDLL, and WinDLL: "The Python global interpreter lock is released before calling any function exported by these libraries, and reacquired afterwards."

CristiFati Over a year ago

@user2357112: Thank you for this fragment! I updated the answer.

TonySalimi Over a year ago

@CristiFati Thanks. Do you khnow if it is the same case for all other packages for loading DLLs in Python, such as Boost.Python?

CristiFati Over a year ago

Not a Boost user, but If what I know is correct, Boost is used to create extension modules (written in C), kind of like Swig does. Being Python modules, they fall under GIL umbrella.

CristiFati Over a year ago

Python code is always meant to run under GIL, one of the design features (and nowadays weaknesses), so it can't be disabled. Pure C code on the other hand shouldn't "benefit" from GIL restriction, therefore the existence of the macros. No one should release the GIL. Is just a feature for bypassing it when needed (in case of parallel operations).

|

Collectives™ on Stack Overflow

Why GIL is not synchrionizing Python threads that are running native C++ code inside a DLL?

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related