https://prefect.io logo
Title
s

Shiyu Gan

04/05/2022, 2:24 AM
In prod, what is the most typical Dask cluster set up? 1. Temporary cluster? 2. Existing cluster? If existing cluster, what is the most popular cluster manager? Yarn, K8s? Thanks!
k

Kevin Kho

04/05/2022, 2:26 AM
Temporary on k8s/fargate or hosted services like Coiled and Saturn Cloud because you can install packages you need easily per flow at runtime
But really it’s upto you. I think Dask recommends long running. I don’t know why, maybe long-running access to the dashboard?
s

Shiyu Gan

04/05/2022, 2:58 AM
Can you link me instruction (from prefect or dask) on temporary on k8s/fargate?
k

Kevin Kho

04/05/2022, 4:29 AM
This has an example of using the KubeCluster