Monitor dask-xarray performance

Question

I have the following basic code which (I thought) should set up xarray to use a LocalCluster.

from dask.distributed import Client
client = Client("tcp://127.0.0.1:46573") # this points to a LocalCluster

import xarray as xr
ds = xr.open_mfdataset('*.nc', combine='by_coords') # Uses dask to defer actually loading data

I now launch some task which also completes with no issues:

(ds.mean('time').mean('longitude')**10).compute()

I noticed that the tabs for the Task graph, Workers, or Task stream (among others) in the dask-labextension for my LocalCluster remain empty. Shouldn't there be some sort of progress displayed while the computation is running?
Which leads me to wonder, how do I tell xarray to explicitly use this cluster? Or is Client a singleton such that xarray only ever has one instance to use anyway?

MRocklin · Accepted Answer · 2020-06-13 15:59:04Z

1

When you create a Dask Client it automatically registers itself as the default way to run Dask computations.

You can check to see if an object is a Dask collection with the dask.is_dask_collection function. As you say, I believe that xr.open_mfdataset uses Dask by default, but this would be a good way to check.

As to why you're not seeing anything on the dashboard, I unfortunately don't know enough about your situation to be able to help you there.

answered Jun 13, 2020 at 15:59

MRocklin

57.5k29 gold badges176 silver badges245 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

doppler Over a year ago

Thank you! As for the dashboard (where I see some things, just not everything I'd thought I would see), I will try to narrow it down before asking for further help.

Collectives™ on Stack Overflow

Monitor dask-xarray performance

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related