How can I run a dask-distributed local cluster from the command line?

Question

I would like to do the equivalent of Client(LocalCluster()) from the command line.

When interacting with distributed from Jupyter notebooks, I end up restarting my kernel often and starting a new LocalCluster each time, as well as refreshing my bokeh webpage.

I would much rather have a process running in the background that I could just connect to, is this possible?

MRocklin · Accepted Answer · 2017-10-07 02:12:14Z

2

The relevant doc page here is http://distributed.readthedocs.io/en/latest/setup.html#using-the-command-line

In one terminal, write the following:

$ dask-scheduler

In another terminal, write the following:

$ dask-worker localhost:8786

The defaults are a bit different here. LocalCluster creates N single-threaded workers while dask-worker starts one N-threaded worker. You can change these defaults with the following keywords

$ dask-worker localhost:8786 --nthreads 1 --nprocs 4

edited Oct 7, 2017 at 2:12

answered Oct 6, 2017 at 17:11

MRocklin

57.5k29 gold badges176 silver badges245 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I run a dask-distributed local cluster from the command line?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related