5

I translated a C++ renderer to Python. The C++ renderer uses threads which each render part of the image. I want to do the same thing in Python. It seems, however, that my multi thread code version takes ages compared to my single thread code version. I am new to multiprocessing in Python and was therefore wondering if the code below actually does what I have in mind: creating a pool of threads, adding and executing some tasks and waiting till completion of all of them?

I know that I cannot compete with my C++ version, but I was hoping to beat the single threaded Python version at least.

Multi thread code

from multiprocessing.pool import ThreadPool

pool = ThreadPool(processes=4)
pool.map(run_task(...), range(11))
pool.close()
pool.join()

Single thread code

for i in range(11):
    Task(...)(i)

Task code

def run_task(...):
    task = Task(...)
    return task.__call__

class Task():
    def __init__(self, ...):
        ...
    def __call__(self, i):
        ...

Edit: I tried to use from multiprocessing import Pool. This seems to block my Python terminal in Canopy IDE. When I run the file from the Windows commandline, I receive:

C:\Users\Matthias\Documents\Courses\Masterproef\pbrt\Tools\Permeability\src>pyth
on renderer.py
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Users\Matthias\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.
2.2785.win-x86_64\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Users\Matthias\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.
2.2785.win-x86_64\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Users\Matthias\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.5.
2.2785.win-x86_64\lib\multiprocessing\pool.py", line 342, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin_
_.instancemethod failed

(This is also why I prefer threads over processes in general. So the GIL design decision makes not really sense to me.)

Improve this question
0

3 Answers 3

Reset to default
7

You should use the process pool instead of the thread pool (see the first example here).

Multithreading should not be used for CPU-bound tasks because of the CPython's GIL.

Maybe this short example will be helpful (let's call it example.py):

from multiprocessing import Pool
import sys

if __name__ == '__main__':

    job_list = [xrange(10000000)]*6

    if 'p' in sys.argv:
        p = Pool(2)
        print("Parallel map")
        print(p.map(sum, job_list))
    else:
        print("Sequential map")
        print(map(sum, job_list))

My machine has 2 cores and example.py p (parallel) version is twice as fast as the sequential one. If we reduce the amount of work to be done (summing ten numbers instead of ten million), the sequential version wins because of the unnecessary overhead of creating processes and delegating tasks in the parallel version.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

I am able to run a small example. But for my full code, I seem to have a pickle problem (what data structure? no idea). Each task receives some references to data structures (like a full scene, which can be gigantic) they all need for reading as well as a camera film they all update. So disabling the GIL (if possible?) would actually solve everything.
In that case, you could try to run your thread-based implementation with something other than CPython. For example, Jython and IronPython have no GIL
I am looking to switch to Visual Studio with IronPython, hopefully this will do the job
Additional info: currently both Jython an IronPython, which have no GIL, don't support numpy. For IronPython, the Ironclad project tries to make IronPython compatible with CPython modules, but is outdated and has no recent status updates. PyPy uses STM instead of (real native) Threads, another way of circumventing the GIL which could for me work as well. Unfortunately PyPy's STM version is only available for Linux distributions.
multiprocessing.Pool is process based and multiprocessing.ThreadPool is thread based?
1

There is no guarantee that multi-threaded python will be faster.

Let alone the overhead of using threads (which generally becomes negligible for 'larger' programs), the Global Interpreter Lock (GIL) means only one thread of actual pure Python will be running. While one thread runs, the others have to wait for it to drop the GIL (e.g. during printing, or a call to some non-python code).

Therefore multi-threaded Python is advantageous if your threaded tasks contain blocking calls that release the GIL, but not guaranteed in general.

Improve this answer

Comments

1

It is important to note that there is a difference between operating systems like Linux and Windows. In Linux the next code works well and utilizes the parallelization:

from multiprocessing.pool import ThreadPool
with ThreadPool(processes=8) as pool:
  result = list(pool.imap(your_func, list_of_items))

But in Windows it just opens many threads which are scheduled in a new process. So the next code worked well for me in Windows as proposed previously:

 from multiprocessing import Pool
 with Pool(8) as pool:
    result = list(pool.map(your_func, list_of_items))

I use Python 3.10.7, Windows 11 Pro

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.