https://prefect.io logo
Title
l

Luke Orland

12/02/2019, 4:15 PM
Are there advantages to executing workflows on a Dask Cluster on Kubernetes vs. on Fargate or other execution environments?
j

josh

12/02/2019, 4:20 PM
It all comes down to preference of your platform of choice for execution and which tool fits best into your workflow! Personally I’m a big fan of Kubernetes w/ a static dask cluster running
l

Luke Orland

12/02/2019, 4:36 PM
Why?
j

josh

12/02/2019, 4:44 PM
K8s provides a nice interface for deploying a dynamic number of Jobs that are easy to maintain, watch, delete, etc… and it’s a tool I’ve been familiar with for a while. It allows you to run the Prefect k8s agent inside your cluster and when it finds flows to run it will kick them off right there in the cluster. Then it has all the added benefits of autoscaling to meet demands. Something like Fargate also provides some nice scaling and modularity but I have much more experience working with k8s!
And a static dask cluster running inside the same k8s cluster is nice because it allows all of your deployed flows to share an internal (non public facing) dask cluster which can be easily scaled!