python multiprocessing with multiple cpus?

Question

I have two Xeon processors (each has 8 cores, so total 32 threads) on my MOBO, and I ran a simple code using multiprocessing.Pool(processors=30). When I monitor using htop, I find that only 12 threads are utilized. Does anyone know why that might be happening?

Are you sure that you are processing more than 12 items (in your map for example)? — grovina
– grovina, Commented Nov 7, 2017 at 1:50
Yes, I had 1080 files (each file is a 3d array of size 0.5-1.5 GB) and I assigned 36 to each process (30 total). In the end it used maybe about 16-20 threads but didn't seem using all requested 30 (I didn't keep monitoring since each process took long time). Maybe some threads were doing IO like Genuine mentioned..? — libphy
– libphy, Commented Nov 8, 2017 at 3:50

Genuine · Accepted Answer · 2017-11-07 01:58:45Z

1

Pool() can provide a specified number of processes for user invocations when a new request is submitted to the Pool, and if the Pool is not full, a new process is created to execute the request; But if the number of processes in the pool has reached the specified maximum, the request will wait until there is a process in the pool to create a new process.
For a model that supports multi-threading, the number of threads recommended is at least 1:1. 5. This allows some threads to do IO.
And if your process doesn't use a full core, it won't take up another core.

answered Nov 7, 2017 at 1:58

Genuine

2641 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

libphy Over a year ago

Thanks. That makes sense.

Collectives™ on Stack Overflow

python multiprocessing with multiple cpus?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related