https://prefect.io logo
k

Kevin Weiler

12/21/2020, 5:45 PM
Is there any way to change executors in the middle of a flow? Use-case in thread -->
I’d like to use Dask for a flow, but prior to running the flow - I’d need to launch a Dask cluster on a more general purpose compute system (Slurm)
so my thought is to have the flow FIRST launch a Dask cluster on Slurm AND THEN run flow on that cluster AND FINALLY tear the cluster down
actually - my thinking on this might be wrong …
the more canonical solution would be to have a Dask Scheduler always running and able to automatically launch workers when a job comes in
k

Kyle Moon-Wright

12/21/2020, 5:57 PM
Hmm, since the executor is always set at the flow level you could also use the orchestrator pattern to separate the instantiation of your cluster from the execution - to send your flow to a cluster created in the first child flow. (just spitballing here šŸ¤”)
k

Kevin Weiler

12/21/2020, 6:00 PM
that would work also - but I think using this is the way to get dynamically sized cluster: https://gateway.dask.org/install-jobqueue.html
šŸš€ 1
šŸ‘ 1
5 Views