Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
s
Shiyu Gan
04/05/2022, 2:24 AMIn prod, what is the most typical Dask cluster set up?
1. Temporary cluster?
2. Existing cluster?
If existing cluster, what is the most popular cluster manager? Yarn, K8s?
Thanks!
k
Kevin Kho
04/05/2022, 2:26 AMTemporary on k8s/fargate or hosted services like Coiled and Saturn Cloud because you can install packages you need easily per flow at runtime
But really it’s upto you. I think Dask recommends long running. I don’t know why, maybe long-running access to the dashboard?
s
Shiyu Gan
04/05/2022, 2:58 AMCan you link me instruction (from prefect or dask) on temporary on k8s/fargate?
k
Kevin Kho
04/05/2022, 4:29 AMThis has an example of using the KubeCluster