Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Is there any way to change executors in the middle of a flow? Use-case in thread --&gt;

I’d like to use Dask for a flow, but prior to running the flow - I’d need to launch a Dask cluster on a more general purpose compute system (Slurm)

so my thought is to have the flow FIRST launch a Dask cluster on Slurm AND THEN run flow on that cluster AND FINALLY tear the cluster down

actually - my thinking on this might be wrong …

the more canonical solution would be to have a Dask Scheduler always running and able to automatically launch workers when a job comes in

Hmm, since the executor is always set at the flow level you could also <https://docs.prefect.io/core/idioms/flow-to-flow.html|use the orchestrator pattern> to separate the instantiation of your cluster from the execution - to send your flow to a cluster created in the first child flow. (just spitballing here :thinking_face:)

that would work also - but I think using this is the way to get dynamically sized cluster:
<https://gateway.dask.org/install-jobqueue.html>