Dask: set multiprocessing method from Python

Question

Is there a way to set the multiprocessing method from Python? I do not see a method in the Client() API docs of Dask.distributed that indicates how to set this property.

Update:

For example, is there:

client = Client(multiprocessing='fork')

or

client = Client(multiprocessing='spawn')

?

MRocklin · Accepted Answer · 2018-06-21 22:51:41Z

4

Unfortunately the multiprocessing context method is set at import time of dask.distributed. If you wanted to set this from Python you could set the config value after you import dask, but before you import dask.distributed.

import dask
dask.config.set({'distributed.worker.multiprocessing-method': 'spawn'})

from dask.distributed import Client

However it's probably more robust to just set this in your config file. See configuration documentation for the various ways to set configuration values.

Note: this is using the configuration as of dask.__version__ == '0.18.0'

answered Jun 21, 2018 at 22:51

MRocklin

57.5k29 gold badges176 silver badges245 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Dask: set multiprocessing method from Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related