Florian Guily
05/25/2022, 1:14 PMlocal dask executor
in this cluster or if there is an aditionnal configuration to use dask as an executor in a k8's cluster ?emre
05/25/2022, 1:55 PMLocalDaskExecutor
is viable and perfectly fine. That will launch a local dask instance within the kubernetes pod your agent launches, and run your flow there. It's a great starting point and sufficient for most use cases.
If you want to scale out to multiple pods/machines, You can either:
⢠Use DaskExecutor
with the url of an already existing dask cluster.
⢠Use something like dask_kubernetes as your cluster_class
. That will launch pods, run dask on them and clean up when your flow finishes. (I think, never used distributed dask for prefect š
)Florian Guily
05/25/2022, 1:57 PM