Short answer: it means the Kubernetes watch your worker is using fell behind and tried to resume with an expired resourceVersion. The API server had already compacted/rotated past that version (due to etcd compaction or lots of updates), so it returns 410 Gone: “too old resource version.” Typical triggers: - High churn in the watched resources (e.g., many Pod/Job updates/deletes) - A long‑lived or briefly disconnected watch that resumes too far behind - API server watch cache/etcd compaction rolling past the requested version This isn’t Prefect-specific and usually isn’t fatal—the client should re-list and re-establish the watch. If you’re seeing it frequently, reduce churn in the namespace, narrow your label selectors, or ensure stable network connectivity for the worker. References: - Kubernetes: Efficient detection of changes - Kubernetes: Watching resources

Thanks for the stack trace — that pinpoints it. Root cause: - Your Prefect Kubernetes worker is “watching” the Kubernetes Job for your flow run. - The watch tries to resume from a stored

resourceVersion

, but the API server has already compacted past that version (etcd compaction / high update churn), so it returns 410 Gone: “Expired: too old resource version”. - This is a normal Kubernetes watch behavior and not a failure of your flow/pod. What it means for your run: - The flow/pod keeps running; this only affects the worker’s ability to monitor via the current watch stream. - Prefect logs “An error occurred while monitoring flow run… not marked as failed”. Monitoring/log streaming/cancellation visibility may be briefly affected until the client re-lists and re-establishes the watch. Why it happens more often: - High churn in the namespace (many Pod/Job updates/deletes) - Long-lived watches or brief network interruptions causing the client to fall behind - API server/etcd compaction advancing the watch cache beyond the stored

resourceVersion

Mitigations you can try: - Narrow the worker’s watch scope with specific label selectors in your work pool’s job template so it only watches the Jobs it creates. - Reduce churn in the namespace (separate busy system components into different namespaces or tune controllers that generate lots of events). - Ensure stable connectivity from the worker to the API server (avoid network interruptions, long GC pauses, etc.). - Upgrade to the latest

prefect

and `prefect-kubernetes`—there have been improvements in watch handling over time. References: - Kubernetes: Efficient detection of changes - Kubernetes: Watching resources - Related Prefect discussion (2.x but same underlying cause): Prefect issue: 410 Expired watch behavior If you’re seeing this frequently, share your Prefect and

prefect-kubernetes

versions and a rough sense of namespace churn—we can suggest more targeted tweaks.

It’s the Kubernetes Job (batch/v1) that Prefect created for your flow run. In your stack trace the error occurs in

prefect_kubernetes.worker._watch_job

while iterating

self._job_events

, which uses a

Watch().stream(...)

against the Job resource. The watch is resuming with an old

resourceVersion

for that Job list, so the API server returns 410 Gone. Notes: - Prefect may also interact with Pods for log streaming, but this particular 410 is from the Job watch. - Job docs: Kubernetes Job