Yes — what you’re describing is consistent with a real weakness in Prefect 3.5.0, not just a mysterious app-level bug. Short version: in 3.5.0, the runner tries to open a cancellation-events websocket during startup, before your flow code runs. If that initial websocket handshake times out, the run can fail outright. There’s no robust retry/fallback at that point in 3.5.0. What I found: What’s happening in 3.5.0 -

FlowRunCancellingObserver

is started as part of runner setup, before user code executes. - That observer opens a websocket subscription to the Prefect API event stream. - The initial connect path goes through

PrefectEventSubscriber.__aenter__

. - In 3.5.0, that initial connection is effectively a single attempt. - The underlying

websockets

client uses an

open_timeout

of 10 seconds by default. - Prefect 3.5.0 does not appear to expose a setting to tune that handshake timeout or disable this startup-time websocket requirement. So if you have an intermittent network-layer issue in Kubernetes — ingress/LB/proxy, DNS, service mesh, NetworkPolicy, node-local egress, idle conn handling, etc. — one bad 10-second handshake can kill the flow run before it starts. That also matches your observation that: - some runs succeed, - some fail before any flow code, - API logs show no incoming attempt from failed jobs. That pattern strongly suggests the connection is being dropped or stalled before it reaches the Prefect API process. This appears improved in later 3.x releases The codebase has later fixes around this area, including: - better handling when the observer websocket cannot be established - retry/backoff behavior for subscriber connection - improved fallback-to-polling behavior when websocket consumption fails So on 3.5.0, your diagnosis is plausible and aligns with how startup worked then. What I’d recommend 1. Upgrade Prefect if you can - This is the strongest recommendation. - If you’re on 3.5.0, moving to a newer 3.x should give you better resilience here. 2. Turn on debug logging for a failing run - Set

PREFECT_DEBUG_MODE=1

- That should help surface the full websocket connection error path. 3. Inspect the network path specifically for websocket support/timeouts Common suspects in self-hosted K8s: - ingress/proxy not reliably handling websocket upgrade - service mesh policy/interception - node/pod DNS slowness - LB timeout/handshake timeout - intermittent egress policy issues - TLS termination/proxy behavior if using

wss://

4. Compare successful vs failing pods Things worth checking: - node placement - namespace/network policy - sidecar injection - DNS resolver behavior - whether failures correlate with specific nodes/AZs 5. Try a direct websocket test from inside a job pod If possible, test connectivity from the same runtime environment to the Prefect events endpoint, not just the REST API endpoint. Why this is probably not your flow code Because the failure occurs in runner setup before flow execution, this is infrastructural/client-startup behavior, not an issue in your flow function itself. Relevant references - Prefect repo: Prefect on GitHub - Pricing / versioning info if needed: prefect.io/pricing If you want, I can help you narrow this down further by looking at: - your ingress/proxy setup, - the exact events websocket URL/path being used, - or the specific Prefect versions where this startup behavior changed.

The main fix appears in Prefect 3.6.0. Based on the code changes I checked: - 3.5.0: initial cancellation-observer websocket connect could fail startup, with no effective retry/fallback at that point - 3.6.0: Prefect added more resilient handling here: - retry/backoff for the initial subscriber connection - graceful fallback behavior instead of killing the run immediately when that observer connection fails There are also additional websocket resiliency improvements after 3.6.0, including fixes in 3.7.0 for cases where the websocket connects but then silently stops delivering events. So if your question is specifically: - “When was the no-retry startup handshake problem fixed?” → 3.6.0 - “When did websocket cancellation observation become more robust overall?” → 3.6.0, with more follow-up hardening in 3.7.0 If you want, I can also map the exact behavior by version, e.g.: - 3.5.x - 3.6.x - 3.7.x and summarize what changed in each.