Dask.distributed only uses one core per node

Question

I have a program where each task is a call to a C++ external program through subprocess.Popen. The tasks are arranged in a graph and everything is executed through the dask get command.

I have a single node version of this program that works just fine with dask.threaded and I am trying to extend this version to a distributed setting. My goal is to run it on a Slurm cluster but I have trouble deploying the workers. When I run the following:

screen -d -m dask-scheduler --scheduler-file scheduler.json
screen -d -m srun dask-worker --scheduler-file scheduler.json
python3 myscript.py

only a single core gets used on every node (out of twenty cores per node).

I did suspect some issues with the GIL but the script works just fine with dask.threaded so I am not quite sure what is going on and some help would be appreciated.

Did you find a solution? I am trying to distribute using a PBS cluster but can only launch one job (to have multiple nodes). Dask is using only cores on only one node. I posted here. — BND
– BND, Commented May 27, 2019 at 6:47

MRocklin · Accepted Answer · 2018-05-18 14:41:40Z

1

I recommend looking at the dashboard to see how many tasks Dask is running at a time on each worker:

Documentation here: http://dask.pydata.org/en/latest/diagnostics-distributed.html

If you see that Dask is only running one task per worker then it's probably a problem in how you've set up your workers (you might want to look at the worker page to get a sense for what Dask thinks you've asked for)

If you see that Dask is running many tasks per worker concurrently then it's probably an issue with your function.

answered May 18, 2018 at 14:41

MRocklin

57.5k29 gold badges176 silver badges245 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Aratz Over a year ago

Didn't really solve the problem but at least I have more insight on what's going on, I'll open a new question with more details, thanks!

Collectives™ on Stack Overflow

Dask.distributed only uses one core per node

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related