How to run a python program that uses numpy on multiple cores, preferably using threading

Question

I want to do parallel computation and return the result to the main thread. Since this is done a lot of times, I assume the overhead of process to process messaging will hinder performance (is this assumption correct?), so I want to use threads.

As I understand - threads will run on different cores only if I'm using jython or ironpython (which is better?).

Assuming this is correct - all I have to do is switch my eclipse interpreter to one of the above?

Finally, I'm using numpy. Is this a problem? Will the implementation of jython/ironpython hinder numpy performance?

Update:

I am now trying to use multiprocesses as per the recommendations below. I'm having trouble passing the arguments in a neat way (also for some reason when I stop the application from running the processes opened do not close and I have to restart my computer!). This is what I'm trying to do:

pool = multiprocessing.Pool()
results = pool.map(my_class(param1=bla1, param2=bla2), list_args)

Where list_args is a list of arguments for __call__ function of the class my_class, and bla1 and bla2 are numpy arrays.

Queries:

The default value for pool is cpu_count(). I assume this is optimal?
Why is this not working? (The processes do not seem to return...)

Have you considered using the multiprocessing module rather than threading? It is pretty easy to use (especially multiprocessing.Pool), and will allow you to use multiple cores for CPU-bound tasks in CPython, since each process will have a separate Global Interpreter Lock. — Blckknght
– Blckknght, Commented Apr 30, 2013 at 3:58

Uri · Accepted Answer · 2013-05-01 10:22:11Z

1

I was missing a main function to wrap everything. Apparently this is important for multiprocessing.

def main():
    run_my_stuff()

if __name__ == "__main__":
    main()

Also - the same effect can be caused if the queue your processes are using is full (it has a certain capacity), so it could help to change the code so that it allows pulling items from the queue at the same time when they are added.

edited May 1, 2013 at 10:22

answered Apr 30, 2013 at 9:54

Uri

27.1k12 gold badges49 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to run a python program that uses numpy on multiple cores, preferably using threading

Update:

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Update:

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related