Hi folks, I am wondering if someone had to deploy a prefect agent and dask-kubernetes execution envi...

Marwan Sarieddine

02/04/2021, 4:00 PM

Hi folks, I am wondering if someone had to deploy a prefect agent and dask-kubernetes execution environment in an istio-enabled namespace The issue we are facing is that the dask-scheduler can no longer find the dask-worker pod to run the tasks so the flow is stuck in a Running state with no tasks executed

Marwan Sarieddine

02/04/2021, 4:02 PM

I recognize this is more of a dask issue but I couldn’t find any recent and relevant issues raised about deploying dask kubernetes with istio

Alvis Tang

02/04/2021, 4:06 PM

Hi Marwan, can you share your config? It sounds like a DNS problem to me. I can run prefect with dask kubernetes, but not in istio thouh. Can you run dask on your cluster?

Marwan Sarieddine

02/04/2021, 4:07 PM

Hi @Alvis Tang - thanks for the response

Can you run dask on your cluster?

Yes I can run dask on the cluster in non-istio enabled namespaces - the issue only arises when istio is enabled

Alvis Tang

02/04/2021, 4:08 PM

hmm I'm not sure, but maybe the envoy sidecar blocks the traffic?

Marwan Sarieddine

02/04/2021, 4:09 PM

hmm I’m not sure, but maybe the envoy sidecar blocks the traffic?

yep that’s probably what I am thinking as well - just wondering how to make envoy allow dask traffic

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.