In prod, what is the most typical Dask cluster set up?
1. Temporary cluster?
2. Existing cluster?
If existing cluster, what is the most popular cluster manager? Yarn, K8s?
Thanks!
k
Kevin Kho
04/05/2022, 2:26 AM
Temporary on k8s/fargate or hosted services like Coiled and Saturn Cloud because you can install packages you need easily per flow at runtime
But really it’s upto you. I think Dask recommends long running. I don’t know why, maybe long-running access to the dashboard?
s
Shiyu Gan
04/05/2022, 2:58 AM
Can you link me instruction (from prefect or dask) on temporary on k8s/fargate?