1

I'm using python 3.3 on a computer with 2 cores but 4 threads. I am trying to learn to use multiprocessing to speed up code, but when using it my code slows down.

To start my learning, I have made a small program:

from multiprocessing import Process
import time

def f():
    s = 0
    for i in range(2*10**7):
        s += i
    return s

if __name__ == '__main__':
    t = time.time()
    p1 = Process(target = f)
    p2 = Process(target = f)
    p3 = Process(target = f)
    p4 = Process(target = f)
    p1.start()
    p2.start()
    p3.start()
    p4.start()
    p1.join()
    p2.join()
    p3.join()
    p4.join()
    print (time.time()-t)

t2 = time.time()
for a in range(4):
    f()
print(time.time()-t2)

Average of 3 runs, the first part with multiprocessing takes 17.15 sec, while the second part without multiprocessing takes 6.24 sec. Using the Windows Task Manager, I see that my computer is indeed using 100% CPU for the first part and only 25% for the second part, and also that I am not running out of memory.

Why is this program so much slower with multiprocessing?

2
  • Is that indentation supposed to change? Commented Jun 14, 2013 at 23:52
  • Yep, I didn't know that the whole program had to be under the if clause. Now it's fixed Commented Jun 15, 2013 at 0:23

1 Answer 1

2

Windows has no fork(), so multiprocessing has to work around that by importing the __main__ module each time a new process is started.

This means when each of your subprocesses runs, it doesn't only run the target function, but also the part at the end of the file. Move it into the block and it should be way faster!

Sign up to request clarification or add additional context in comments.

4 Comments

This worked, thanks! Now the time for part with multiprocessing 3.44 sec. This is almost a 2x speed up (from without multiprocessing), but I still expected slightly more. Is there anyway to get closer to 4x speed up?
no, that's as close as you'll get for cpu bound problems. your cpu has 2 cores, which means that still only two threads can be active at the same time, having 4 core threads basically just reduces some overhead when switching threads.
Ok, so I have 2 questions to make sure I understand this correctly. First, when a regular python program is running without multiprocessing and task manager says my computer is only using 25% of the CPU, is my computer actually using about 50% of the processing power? Second, if I ran only 2 processes, but had each do twice the work, would the program take about the same amount of time as 4 processes?
1) - i don't really know what the windows taskmanager shows as cpu usage, I've found some differing interpretations. This article seems to give a good idea. 2) probably, seems about right. just try it :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.