Hi all, I'm running a flow on Kubernetes using DaskExecutor with 4 workers, but it doesn't create 4 worker pods. Ideally, I'd like to be able to scale to >1 node (not using an existing Dask cluster). (I tried using dask_kubernetes.KubeCluster, but I get an error: AttributeError("'NoneType' object has no attribute 'metadata'"). I'm not sure if this would solve my problem anyway.) Is there a way I can run a flow such that I can scale up to multiple nodes? Thank you

Hi @Nikul using the dask executor doesn’t automatically scale up nodes in your k8s cluster. Using the dask_kubernetes package would be a way to accomplish this! If you’re encountering an error in using it with your flow I recommend opening an issue on the repo with a reproducible example so we can triage and see what might be happening 🤔