How to use `client.start_ipython_workers()` in dask-distributed?

Question

I am trying to get workers to output some information from their ipython kernel and execute various commands in the ipython session. I tried the examples in the documentation and the ipyparallel example works, but not the second example (with ipython magics). I cannot get the workers to execute any commands. For example, I am stuck on the following issue:

from dask.distributed import Client

client = Client()

info = client.start_ipython_workers()

list_workers = info.keys()

%remote info[list_workers[0]]

The last line returns an error:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-19-9118451af441> in <module>
----> 1 get_ipython().run_line_magic('remote', "info['tcp://127.0.0.1:50497'] worker.active")

~/miniconda/envs/dask/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
   2334                 kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2335             with self.builtin_trap:
-> 2336                 result = fn(*args, **kwargs)
   2337             return result
   2338 

~/miniconda/envs/dask/lib/python3.7/site-packages/distributed/_ipython_utils.py in remote_magic(line, cell)
    115     info_name = split_line[0]
    116     if info_name not in ip.user_ns:
--> 117         raise NameError(info_name)
    118     connection_info = dict(ip.user_ns[info_name])
    119 

NameError: info['tcp://127.0.0.1:50497']

I would appreciate any examples of how to get any information from the ipython kernel running on workers.

if you first do worker=info[list_workers[0]], and then %remote worker ..., you get one step further, but I suspect things still don't work. — mdurant
– mdurant, Commented Feb 19, 2021 at 17:13
Thank you for commenting! Yes, I tried that and also tried registering custom magic with magic_names option in start_ipython_workers. I can see that the magics are created but when I try to use them I get TimeoutError: Timeout waiting for IPython output. — SultanOrazbayev
– SultanOrazbayev, Commented Feb 19, 2021 at 17:31
Please raise an issue, if there isn't one already. It may be that this functionality has been abandoned. — mdurant
– mdurant, Commented Feb 19, 2021 at 17:42
Ah, ok, will do, although this is more of a curiosity for me. — SultanOrazbayev
– SultanOrazbayev, Commented Feb 19, 2021 at 17:43

SultanOrazbayev · Accepted Answer · 2021-02-19 17:52:38Z

0

Posting here just for keeping track, I raised an issue for this on GitHub: https://github.com/dask/distributed/issues/4522

answered Feb 19, 2021 at 17:52

SultanOrazbayev

16.7k3 gold badges25 silver badges59 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use `client.start_ipython_workers()` in dask-distributed?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related