Hey! I'm working with Prefect in Azure using K8s, ACR and Prefect Cloud. Is there a setup for affinity or taints/tolerations? I'm trying to put tolerations in the KubernetesRun template and the DaskExecutor as well. It works for tolerations, but on the DaskExecutor, it gives this error:

RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS.

Does anyone know about this error with K8s tolerations?

Do you work with Thomas here ?

Well yeah, pretty much same error, not going anywhere from this.

I know I didn’t get a chance to explore yet. Really weird issue

Indeed weird! I'm almost sure that there is another taint over the scheduler<->worker for dask... but I really don't know where to debug it since it's a dask_kubernetes thing. Since the traceback has errors with port_forward, that might be the case...

I really can’t find any info on this. I would say the Dask discourse might be able to help a lot better

@Kevin Kho Found the issue with the error. Somehow the dask-kubernetes 2022.5.2 and 2022.5.1 are breaking prefect's DaskExecutor (I saw that they changed a few lines). Only 2022.5.0 or below are working.

That’s perfect! Thanks for digging in. @Marvin “Dask Kubernetes greater than 2022.5.0 has issues with DaskExecutor”

@Marvin open "The dask-kubernetes package greater than 2022.5.0 causes issues with DaskExecutor"

Oof! lol

Hi all — our team is experiencing the same issue. Is there a recommended workaround for this in the meantime?

Yeah. go down to

dask_kubernetes

0.5.0

Got it, thank you Kevin!