How do I run a group of nodes together with Dask

Question

I have an image processing graph and I want to process many images in batch. My graph looks like the following:

When I run the graph bokeh shows the execution path like this:

This causes my machines to run out of memory and crash as the output of the load image is megabytes of images. I would like the graph to run like this as as the result of Save result is very small and should be fine:

How can I do this with dask?

Customizing Optimization seems like it would be useful where I could possibly fuse the middle nodes. Is this the best way?

mdurant · Accepted Answer · 2019-03-13 13:22:26Z

1

Dask prefers to execute tasks where memory can be freed, which ought to be depth-wise in your example. However, it also gives you parallelism; so the easiest way may be just to have one worker.

Indeed, the linear chains in the graph would make a good case for fuse. You can call optimise yourself (dask.optimization.inline_functions, dask.optimization.fuse, shouldn't need custom), or you could write a function which explicitly calls each sub-task in turn within a single task (save(process(load(..)))).

answered Mar 13, 2019 at 13:22

mdurant

28.8k5 gold badges49 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Matt Nicolls Over a year ago

Graph ended up not looking like I thought in the graph tab so when I updated it did as you said and ran depth first, thanks!

Collectives™ on Stack Overflow

How do I run a group of nodes together with Dask

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related