Running shell commands in parallel using dask distributed

Question

I have a folder with a lot of .sh scripts. How can I use an already set up dask distributed cluster to run them in parallel?

Currently, I am doing the following:

import dask, distributed, os

# list with shell commands that I want to run
commands = ['./script1.sh', './script2.sh', './script3.sh']

# delayed function used to execute a command on a worker
run_func = dask.delayed(os.system)

# connect to cluster
c = distributed.Client('my_server:8786')

# submit job
futures = c.compute( [run_func(c) for c in commands])

# keep connection alive, do not exit python
import time
while True:
    time.sleep(1)

This works, however for this scenario it would be ideal if the client could disconnect without causing the scheduler to cancel the job. I am looking for a way to compute my tasks that does not require an active client connection. How could this be done?

mdurant · Accepted Answer · 2018-04-07 19:50:55Z

4

Have you seen http://distributed.readthedocs.io/en/latest/api.html#distributed.client.fire_and_forget ? That would be a way to ensure that some task runs on the cluster after the client has gone.

Note also that you have functions like wait() or even gather() so you don't need sleep-forever loops.

In general, though, subprocess.Popen will launch a child process and not wait for it to finish, so you don't even need anything complex from dask, since it doesn't appear you are interested in any output from the call.

answered Apr 7, 2018 at 19:50

mdurant

28.8k5 gold badges49 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Arco Bast Over a year ago

I just figured out fire_and_forget is also nicely documented here: dask.pydata.org/en/latest/futures.html#fire-and-forget

Collectives™ on Stack Overflow

Running shell commands in parallel using dask distributed

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related