Hi all, I'm running a flow on Kubernetes using DaskExecutor with 4 workers, but it doesn't create 4 worker pods. Ideally, I'd like to be able to scale to >1 node (not using an existing Dask cluster).
(I tried using dask_kubernetes.KubeCluster, but I get an error: AttributeError("'NoneType' object has no attribute 'metadata'"). I'm not sure if this would solve my problem anyway.)
Is there a way I can run a flow such that I can scale up to multiple nodes?
Thank you