Raising multiple exceptions from threads in Python

Question

tl;dr I spawn 3 threads, each thread throws an exception, most pythonic way to raise all 3 exceptions?

Below is a code example that is similar to what I am doing.

from multiprocessing.pool import ThreadPool

def fail_func(host):
    raise Exception('{} FAILED!!!'.format(host))

hosts = ['172.1.1.1', '172.1.1.2', '172.1.1.3']
pool = ThreadPool(processes=5)
workers = [pool.apply_async(fail_func(host)) for host in hosts]
# join and close thread pool
pool.join(); pool.close()
# get the exceptions
[worker.get() for worker in workers if not worker.successful()]

What it ends up doing is just failing on the 1st host with the following traceback:

Traceback (most recent call last):
  File "thread_exception_example.py", line 8, in <module>
    workers = [pool.apply_async(fail_func(host)) for host in hosts]
  File "thread_exception_example.py", line 4, in fail_func
    raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.1 FAILED!!!

But what I want it to do is raise multiple exceptions for each thread that failed, like so:

Traceback (most recent call last):
  File "thread_exception_example.py", line 8, in <module>
    workers = [pool.apply_async(fail_func(host)) for host in hosts]
  File "thread_exception_example.py", line 4, in fail_func
    raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.1 FAILED!!!

Traceback (most recent call last):
  File "thread_exception_example.py", line 8, in <module>
    workers = [pool.apply_async(fail_func(host)) for host in hosts]
  File "thread_exception_example.py", line 4, in fail_func
    raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.2 FAILED!!!

Traceback (most recent call last):
  File "thread_exception_example.py", line 8, in <module>
    workers = [pool.apply_async(fail_func(host)) for host in hosts]
  File "thread_exception_example.py", line 4, in fail_func
    raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.3 FAILED!!!

is there any pythonic way of doing this? or do I need to wrap everything in a try/except, collect all the messages, then re-raise a single Exception?

As a side note, it's a little confusing to call your results objects worker. Normally, that word refers to the subprocesses in the pool. — abarnert
– abarnert, Commented Aug 16, 2013 at 22:41
As another side note, multiprocessing.pool.ThreadPool is an undocumented feature, while multiprocessing.dummy.Pool is documented and should give you what you want. I personally think that in this case, the much clearer name ThreadPool vs. dummy.Pool outweighs that, but it's worth knowing the choice for yourself. — abarnert
– abarnert, Commented Aug 16, 2013 at 22:47

abarnert · Accepted Answer · 2013-08-16 22:44:37Z

2

There is no way to "raise multiple exceptions". In a given exception context, there is either an exception, or not.

So yes, you will have to create a wrapper exception that holds all of the exceptions, and raise that. But you've almost got all the code you need:

def get_exception():
    try:
        worker.get()
    except Exception as e:
        return e

Now, instead of:

[worker.get() for worker in workers if not worker.successful()]

… you can just do:

[get_exception(worker.get) for worker in workers if not worker.successful()]

And that's a list of exceptions.

Personally, I've always thought AsyncResult should have an exception method, similar to the one in concurrent.futures.Future. But then I would have used futures here in the first place (installing the backport if I were forced to use Python 2.x).

edited Aug 16, 2013 at 22:44

answered Aug 16, 2013 at 22:38

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Raising multiple exceptions from threads in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related