Hi all, I have a question about the Prefect Kubernetes integration. We are running a Kubernetes work pool (Prefect v3.4.19) and everything is generally working fine - except for a couple of (possibly related) issues: Jobs Stuck in Running We are hitting Job State Stuck in Running After Pod Eviction https://github.com/PrefectHQ/prefect/issues/15945 • The workaround described in related issue did work but it seems like a total hack: https://github.com/PrefectHQ/prefect/issues/15408 No Kubernetes Events When running

prefect event stream

, I don't see any events named like

prefect.kubernetes.pod

as we might expect to see from the observer code Some setup details - we are running 3x servers, 3x workers, with redis. Has anyone previously reported similar problems? We are intending to get set up with a dev environment to hack/debug this directly but wanted to ask here first.

hi @Sebastian S

No Kubernetes Events

this seems like it could be its own issue or discussion, are you open to creating one? if we identify that there's something up with the worker/observer impl (and not a network/config issue blocking websockets required for events) then it would be good to have that articulated so we can open a PR

Yup sure ! Are discussions the primary place to engage with prefect core dev directly?

broadly speaking, ideally • discussions are for "id like to know how to X" or "i think Y is broke but not sure if due to A or B" • issues are for "i can empirically show X which is problematic because Y" or "i am confident Z is a good enhancement" both are more discoverable than slack. discussions feed into well-scoped issues etc

Thanks!

sounds great! thanks

I'm facing similar issues within an EKS cluster. Prefect Server + 2 workers deployed with their Helm chart. We can spawn the flows but then we totally loose control over them. If it crashes for OOM/Error, it keeps in "Running" state, and if we Cancel it from the web UI, the Job Pod keeps running and the workers or server logs say nothing.