Is there any way to change executors in the middle of a flow

Question

Is there any way to change executors in the middle of a flow Use case in thread gt

Kevin Weiler · Answer

I d like to use Dask for a flow but prior to running the flow I d need to launch a Dask cluster on a more general purpose compute system Slurm

Kevin Weiler · Answer

so my thought is to have the flow FIRST launch a Dask cluster on Slurm AND THEN run flow on that cluster AND FINALLY tear the cluster down

Kevin Weiler · Answer

actually my thinking on this might be wrong

Kevin Weiler · Answer

the more canonical solution would be to have a Dask Scheduler always running and able to automatically launch workers when a job comes in

Kyle Moon-Wright · Answer

Hmm since the executor is always set at the flow level you could also <https docs prefect io core idioms flow to flow html|use the orchestrator pattern> to separate the instantiation of your cluster from the execution to send your flow to a cluster created in the first child flow just spitballing here thinking face

Kevin Weiler · Answer

that would work also but I think using this is the way to get dynamically sized cluster <https gateway dask org install jobqueue html>