In prod, what is the most typical Dask cluster set...
# prefect-community
s
In prod, what is the most typical Dask cluster set up? 1. Temporary cluster? 2. Existing cluster? If existing cluster, what is the most popular cluster manager? Yarn, K8s? Thanks!
k
Temporary on k8s/fargate or hosted services like Coiled and Saturn Cloud because you can install packages you need easily per flow at runtime
But really it’s upto you. I think Dask recommends long running. I don’t know why, maybe long-running access to the dashboard?
s
Can you link me instruction (from prefect or dask) on temporary on k8s/fargate?
k
This has an example of using the KubeCluster