Dask Distributed: Getting some errors after computations

Question

I am running Dask Distributed on Linux CentOS 7, with a Python 3.6.2 installation. My computation seems to be getting fine (I am still improving my code, but I am able to have some results), but I keep getting some python errors apparently linked to tornado module. I am only launching a one node standalone Dask distributed cluster.

Here is the most common example:

Exception in thread Client loop:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 832, in start
    self._run_callback(self._callbacks.popleft())
AttributeError: 'NoneType' object has no attribute 'popleft'

And here is another one:

tornado.application - ERROR - Exception in callback <bound method WorkStealing.balance of <distributed.stealing.WorkStealing object at 0x7f752ce6d6a0>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 1026, in _run
    return self.callback()
  File "/usr/local/lib/python3.6/site-packages/distributed/stealing.py", line 248, in balance
    sat = s.rprocessing[key]
KeyError: 'read-block-9024000000-e3fefd2110094168cc0505db69b326e0'

Do you have any idea why? Should I close some connections or stop the standalone cluster?

OK, right after posting my question, it seems that simply adding client.close() at the end of my code get rid of those exceptions. Maybe this should be stated in the documentation, I could not find it (but I can be kind of blind sometimes). — Guillaume EB
– Guillaume EB, Commented Aug 21, 2017 at 15:44

MRocklin · Accepted Answer · 2017-08-23 12:23:43Z

1

Yes, if you don't close down the Tornado IOLoop before exiting the process then it can die in an unpleasant way. Fortunately this shouldn't affect your application, except by looking unpleasant.

You might submit a bug report about this, it's still something that we should fix.

answered Aug 23, 2017 at 12:23

MRocklin

57.5k29 gold badges176 silver badges245 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

keflavich Over a year ago

Is there an associated bug report / issue we can look at to see if this is resolved?

Collectives™ on Stack Overflow

Dask Distributed: Getting some errors after computations

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related