Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
l
Luke Orland
12/02/2019, 4:15 PMAre there advantages to executing workflows on a Dask Cluster on Kubernetes vs. on Fargate or other execution environments?
j
josh
12/02/2019, 4:20 PMIt all comes down to preference of your platform of choice for execution and which tool fits best into your workflow! Personally I’m a big fan of Kubernetes w/ a static dask cluster running
l
Luke Orland
12/02/2019, 4:36 PMWhy?
j
josh
12/02/2019, 4:44 PMK8s provides a nice interface for deploying a dynamic number of Jobs that are easy to maintain, watch, delete, etc… and it’s a tool I’ve been familiar with for a while. It allows you to run the Prefect k8s agent inside your cluster and when it finds flows to run it will kick them off right there in the cluster. Then it has all the added benefits of autoscaling to meet demands.
Something like Fargate also provides some nice scaling and modularity but I have much more experience working with k8s!
And a static dask cluster running inside the same k8s cluster is nice because it allows all of your deployed flows to share an internal (non public facing) dask cluster which can be easily scaled!