https://prefect.io logo
Title
l

Lucas Hosoya

05/31/2022, 2:15 PM
Hey! I'm working with Prefect in Azure using K8s, ACR and Prefect Cloud. Is there a setup for affinity or taints/tolerations? I'm trying to put tolerations in the KubernetesRun template and the DaskExecutor as well. It works for tolerations, but on the DaskExecutor, it gives this error:
RuntimeError: Missing dependency kubectl. Please install kubectl following the instructions for your OS.
Does anyone know about this error with K8s tolerations?
k

Kevin Kho

05/31/2022, 2:19 PM
Do you work with Thomas here ?
l

Lucas Hosoya

05/31/2022, 2:20 PM
Well yeah, pretty much same error, not going anywhere from this.
k

Kevin Kho

05/31/2022, 2:22 PM
I know I didn’t get a chance to explore yet. Really weird issue
l

Lucas Hosoya

05/31/2022, 2:24 PM
Indeed weird! I'm almost sure that there is another taint over the scheduler<->worker for dask... but I really don't know where to debug it since it's a dask_kubernetes thing. Since the traceback has errors with port_forward, that might be the case...
k

Kevin Kho

05/31/2022, 2:41 PM
I really can’t find any info on this. I would say the Dask discourse might be able to help a lot better
👀 1
l

Lucas Hosoya

05/31/2022, 6:12 PM
@Kevin Kho Found the issue with the error. Somehow the dask-kubernetes 2022.5.2 and 2022.5.1 are breaking prefect's DaskExecutor (I saw that they changed a few lines). Only 2022.5.0 or below are working.
k

Kevin Kho

05/31/2022, 6:16 PM
That’s perfect! Thanks for digging in. @Marvin “Dask Kubernetes greater than 2022.5.0 has issues with DaskExecutor”
a

Anna Geller

05/31/2022, 9:02 PM
@Marvin open "The dask-kubernetes package greater than 2022.5.0 causes issues with DaskExecutor"
k

Kevin Kho

05/31/2022, 9:04 PM
Oof! lol
i

Ilya Galperin

06/01/2022, 5:12 PM
Hi all — our team is experiencing the same issue. Is there a recommended workaround for this in the meantime?
k

Kevin Kho

06/01/2022, 5:14 PM
Yeah. go down to
dask_kubernetes
0.5.0
I mean 2022.5.0
i

Ilya Galperin

06/01/2022, 5:26 PM
Got it, thank you Kevin!