Kevin Weiler

12/21/2020, 5:45 PM
Is there any way to change executors in the middle of a flow? Use-case in thread -->
Iā€™d like to use Dask for a flow, but prior to running the flow - Iā€™d need to launch a Dask cluster on a more general purpose compute system (Slurm)
so my thought is to have the flow FIRST launch a Dask cluster on Slurm AND THEN run flow on that cluster AND FINALLY tear the cluster down
actually - my thinking on this might be wrong ā€¦
the more canonical solution would be to have a Dask Scheduler always running and able to automatically launch workers when a job comes in

Kyle Moon-Wright

12/21/2020, 5:57 PM
Hmm, since the executor is always set at the flow level you could also use the orchestrator pattern to separate the instantiation of your cluster from the execution - to send your flow to a cluster created in the first child flow. (just spitballing here šŸ¤”)

Kevin Weiler

12/21/2020, 6:00 PM
that would work also - but I think using this is the way to get dynamically sized cluster:
šŸš€ 1
šŸ‘ 1