1

I am trying to use multiprocessing to return a list, but instead of waiting until all processes are done, I get several returns from one return statement in mp_factorizer, like this:

None
None
(returns list)

in this example I used 2 threads. If I used 5 threads, there would be 5 None returns before the list is being put out. Here is the code: 

def mp_factorizer(nums, nprocs, objecttouse):
    if __name__ == '__main__':
        out_q = multiprocessing.Queue()
        chunksize = int(math.ceil(len(nums) / float(nprocs)))
        procs = []
        for i in range(nprocs):
            p = multiprocessing.Process(
                    target=worker,                   
                    args=(nums[chunksize * i:chunksize * (i + 1)],
                          out_q,
                    objecttouse))
            procs.append(p)
            p.start()

        # Collect all results into a single result dict. We know how many dicts
        # with results to expect.
        resultlist = []
        for i in range(nprocs):
            temp=out_q.get()
            index =0
            for i in temp:
                resultlist.append(temp[index][0][0:])
                index +=1

        # Wait for all worker processes to finish
        for p in procs:
            p.join()
            resultlist2 = [x for x in resultlist if x != []]
        return resultlist2

def worker(nums, out_q, objecttouse):
    """ The worker function, invoked in a process. 'nums' is a
        list of numbers to factor. The results are placed in
        a dictionary that's pushed to a queue.
    """
    outlist = []
    for n in nums:        
        outputlist=objecttouse.getevents(n)
        if outputlist:
            outlist.append(outputlist)   
    out_q.put(outlist)

mp_factorizer gets a list of items, # of threads, and an object that the worker should use, it then splits up the list of items so all threads get an equal amount of the list, and starts the workers. The workers then use the object to calculate something from the given list, add the result to the queue. Mp_factorizer is supposed to collect all results from the queue, merge them to one large list and return that list. However - I get multiple returns.

What am I doing wrong? Or is this expected behavior due to the strange way windows handles multiprocessing? (Python 2.7.3, Windows7 64bit)

EDIT: The problem was the wrong placement of if __name__ == '__main__':. I found out while working on another problem, see using multiprocessing in a sub process for a complete explanation.

Improve this question
2
  • What do you mean by "I get several returns from one return statement"? Also, please post a runnable example. Commented Jan 29, 2013 at 14:29
  • I'll post an example, but it'll take a while to make sure the problem gets reproduced by the example. By several returns I mean that if I would write print mp_factorizer(list, 2, someobject), I would not get the print command executed once,but once plus as many times as I have set the number of threads to, i.e. with 2 threads: None None (prints list) (each in a new line) print should only get executed once. But I would get 3 printouts. So in fact the return statement would be executed every time a worker finishes(?) and at the end, when the list gets returned. Commented Jan 29, 2013 at 15:22

2 Answers 2

Reset to default
1

if __name__ == '__main__' is in the wrong place. A quick fix would be to protect only the call to mp_factorizer like Janne Karila suggested:

if __name__ == '__main__':
    print mp_factorizer(list, 2, someobject)

However, on windows the main file will be executed once on execution + once for every worker thread, in this case 2. So this would be a total of 3 executions of the main thread, excluding the protected part of the code.

This can cause problems as soon as there are other computations being made in the same main thread, and at the very least unnecessarily slow down performance. Even though only the worker function should be executed several times, in windows everything will be executed thats not protected by if __name__ == '__main__'.

So the solution would be to protect the whole main process by executing all code only after if __name__ == '__main__'.

If the worker function is in the same file, however, it needs to be excluded from this if statement because otherwise it can not be called several times for multiprocessing.

Pseudocode main thread:

# Import stuff
if __name__ == '__main__':
    #execute whatever you want, it will only be executed 
    #as often as you intend it to
    #execute the function that starts multiprocessing, 
    #in this case mp_factorizer()
    #there is no worker function code here, it's in another file.

Even though the whole main process is protected, the worker function can still be started, as long as it is in another file.

Pseudocode main thread, with worker function:

# Import stuff
#If the worker code is in the main thread, exclude it from the if statement:
def worker():
    #worker code
if __name__ == '__main__':
    #execute whatever you want, it will only be executed 
    #as often as you intend it to
    #execute the function that starts multiprocessing, 
    #in this case mp_factorizer()
#All code outside of the if statement will be executed multiple times
#depending on the # of assigned worker threads.

For a longer explanation with runnable code, see using multiprocessing in a sub process

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

Your if __name__ == '__main__' statement is in the wrong place. Put it around the print statement to prevent the subprocesses from executing that line:

if __name__ == '__main__':
    print mp_factorizer(list, 2, someobject)

Now you have the if inside mp_factorizer, which makes the function return None when called inside a subprocess.

Improve this answer

2 Comments

Exactly. I just found out myself yesterday too. However, just protecting the part that calls mp_factorizer will be no good idea, since python will execute all code in the main process that is not being protected by if __name__ == '__main__', as many times as there are threads being launched, in this case 2 times too often. See my answer for details.
@harbun I would not worry much about executing one extra function definition. But in principle you are correct, and a real app can do some heavy initialization that you may want to avoid in the subprocesses.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.