how to skip error in python multiprocessing

Question

I use to run long independent jobs with multiprocessing.Pool.map

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count())
input_var = [1,2,3]
ris = pool.map(long_function,input_var)
pool.close()
pool.join()

This works well but if for example I get an error in long_function(2) I will lose all the information that I have obtained with long_function(1) and long_function(3).

is there a way to avoid this?

The best would be to obtain an output like ris=[long_function(1), ERROR, long_function(3)] Is there anyway to do that?

johntellsall · Accepted Answer · 2014-07-15 02:12:54Z

2

def safe_long_function(*args, **kwargs):
    try:
        return long_function(*args, **kwargs)
    except Exception as e:
        return e

You basically want to catch the exceptions thrown and then return them rather than raise them. For example

def long_function(x):
    if x == 2:
        raise Exception("This number is even")

import multiprocessing
pool = multiprocessing.Pool() # default is num CPUs
input_var = [1,2,3]
ris = pool.map(safe_long_function, input_var)
pool.close()
pool.join()

print ris

This will give [1, Exception("This number is even"), 3]

You can then do something like

for result in ris:
    if isinstance(result, Exception):
        print "Error: %s" % result
    else:
        print result

edited Jul 15, 2014 at 2:12

johntellsall

15.2k4 gold badges48 silver badges40 bronze badges

answered Jul 10, 2014 at 18:41

Jakob Bowyer

34.8k8 gold badges80 silver badges93 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how to skip error in python multiprocessing

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related