Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Are there advantages to executing workflows on a Dask Cluster on Kubernetes vs. on Fargate or other execution environments?

related: <https://docs.prefect.io/cloud/recipes/k8s_dask.html>

It all comes down to preference of your platform of choice for execution and which tool fits best into your workflow! Personally I’m a big fan of Kubernetes w/ a static dask cluster running

K8s provides a nice interface for deploying a dynamic number of Jobs that are easy to maintain, watch, delete, etc… and it’s a tool I’ve been familiar with for a while. It allows you to run the Prefect k8s agent inside your cluster and when it finds flows to run it will kick them off right there in the cluster. Then it has all the added benefits of autoscaling to meet demands.

Something like Fargate also provides some nice scaling and modularity but I have much more experience working with k8s!

And a static dask cluster running inside the same k8s cluster is nice because it allows all of your deployed flows to share an internal (non public facing) dask cluster which can be easily scaled!