Florian Guily05/25/2022, 1:14 PM
in this cluster or if there is an aditionnal configuration to use dask as an executor in a k8's cluster ?
local dask executor
emre05/25/2022, 1:55 PM
is viable and perfectly fine. That will launch a local dask instance within the kubernetes pod your agent launches, and run your flow there. It's a great starting point and sufficient for most use cases.
If you want to scale out to multiple pods/machines, You can either:
with the url of an already existing dask cluster.
• Use something like dask_kubernetes as your
. That will launch pods, run dask on them and clean up when your flow finishes. (I think, never used distributed dask for prefect 😅)
Florian Guily05/25/2022, 1:57 PM