python multiprocessing: processes don't work

Question

I want to assign my computing jobs to more than one cpu, so I choose multiprocessing. However, the result is not what I want.

import numpy as np
from multiprocessing import Process

def func(begin,end):
    print('*'*5)
    print('begin=%d' %(begin))
    for i in range(begin,end):
        for j in range(10):
            myarray[i][j]=1

myarray=np.zeros((12,10))
print(myarray)
for i in range(4):
    begin=i*3
    end=(i+1)*3
    p=Process(target=func,args=(begin,end,))
    p.start()

print('*'*5)
print(myarray)

I think that myarray should be all ones. But it doesn't change at all. Why? The func function does not change elements of myarray? I tried the example from this linkenter link description here

from multiprocessing import Process
def f(name):
    print('hello',name)

p=Process(target=f,args=('bob',))
p.start()

It shows nothing on the screen. Why? How should I finish my computation with python? Can anyone give a way of fully take advantage of multi-cpus?

@alfasin Seems to me it should. But I test that code using python 3.4.3 on win7 and corresponding code(the print function to statement) with python 2.6.6 on centos6.5 in virtual box. Neither show anything. — dudu
– dudu, Commented Aug 31, 2015 at 4:15
@alfasin Sorry. The second didn't work on win7 with python3.4.3. This is the platform I use to test simple programs. But it is okay with centos with python 2.6.6. I've checked that again. Although I cann't figure out why, it is the situation now. — dudu
– dudu, Commented Aug 31, 2015 at 6:54

DWilches · Accepted Answer · 2015-08-31 03:59:50Z

2

There are two problems there:

When you print your array at the end, how do you know your processes have already finished? You need to invoke join() on each process to ensure they have finished.
Each process has a copy of "myarray". If you want to communicate several processes you need to either use a Queue or a Pipe. Check the documentation that talks about exchanging data between processes

Here is a working example using as base the one you posted (it is not intended to be fast, just to show how the communication is done):

from multiprocessing import Process, freeze_support, Queue

def func(my_id, q, begin, end):
    global myarray
    print('Process %d has range: %d - %d' % (my_id, begin, end))
    for i in range(begin,end):
        q.put((i, i * 2))

if __name__ == "__main__":
    freeze_support()

    q = Queue()
    processes = []
    myarray=[0] * 12
    print("At the beginning the array is ", myarray)

    for i in range(4):
        begin = i*3
        end = (i+1)*3
        p = Process(target=func, args=(i, q, begin, end))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

    while not q.empty():
        (index, value) = q.get()
        myarray[index] = value

    print("At the end the array is ", myarray)

Try to change the line p.join() by pass to see what happens :)

edited Aug 31, 2015 at 3:59

answered Aug 31, 2015 at 3:53

DWilches

23.4k17 gold badges82 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dudu Over a year ago

Yes, join is a must. If I replace the p.join() with pass, some values are changed while others are not.

peihuadeng · Accepted Answer · 2015-08-31 05:40:57Z

That is because the print(myarray) statement execute before the process end. You should wait for the process end and then execute this statement. Please note the thread.join() statement at the following.

import numpy as np
import threading

lock = threading.RLock()
thread_list = []
def func(begin,end):
    print('*'*5)
    print('begin=%d' %(begin))
    for i in range(begin,end):
        for j in range(10):
            with lock:
                myarray[i][j]=1

myarray=np.zeros((12,10))
print(myarray)
for i in range(4):
    begin=i*3
    end=(i+1)*3
    p=threading.Thread(target=func,args=(begin,end,))
    p.start()
    thread_list.append(p)

print('*'*5)
for thread in thread_list:
    thread.join()
print(myarray)

Collectives™ on Stack Overflow

python multiprocessing: processes don't work

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related