What’s happening is: Your flow run has a deployment concurrency lease. Prefect renews that lease while the run is executing. If renewal fails and Prefect learns the lease is gone (

410 Gone

), it cancels the run on purpose to avoid exceeding the deployment’s concurrency limit. So this message is expected behavior for that failure mode:

Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.

And this is the key part:

410 Gone ... Lease not found - it may have expired or been revoked

That means the server no longer thinks your run owns the concurrency slot. Why this happens In Prefect 3.x, deployment concurrency uses a lease that must be periodically renewed. A renewal can fail because of things like: - temporary loss of connectivity between the flow-run pod and the Prefect API/server - the lease expiring before it could be renewed - the lease being revoked/cleaned up server-side - server instability/restarts around lease state For deployment concurrency specifically, Prefect is strict here: if the lease is lost, it cancels execution rather than risking multiple runs exceeding the configured limit. So the

Execution was cancelled by the runtime environment.

error is basically the runtime saying: “I no longer hold the reserved slot, so I must stop.” --- About the pod still running Ideally, when the run is cancelled, the Kubernetes worker should delete the Kubernetes Job, which should terminate the pod too. If the pod keeps running, there are a few possibilities: - the worker did not successfully process the cancellation - the worker lost track of the job - Kubernetes is still in termination/grace-period handling - the process inside the container is not stopping promptly after cancellation - there is some worker/infra edge case or bug So: the flow being cancelled in Prefect and the pod still consuming RAM is not the intended end state. The run cancellation is expected; the orphaned/still-running pod is the suspicious part. --- Why deleting the K8s Job retried the flow run That part is also explainable. If you manually delete the Kubernetes Job/pod for an active flow run, the worker/orchestration layer may interpret that as infrastructure failure rather than a normal completed cancellation path. If your deployment has retries configured, Prefect can create another attempt. So deleting the job from Kubernetes is not the safest way to stop these runs if retries are enabled. --- How to handle this 1. Investigate why the lease renewal is being lost This is the main issue. Check for: - connectivity from flow-run pods to

prefect-server.prefect.svc.cluster.local:4200

- restarts of the Prefect server/API around the same timestamps - network policies / service mesh / DNS instability - long GC pauses / CPU starvation in the run pod or server - clock skew is less likely, but worth checking in badly skewed clusters Since your URL is in-cluster, I’d especially suspect: - intermittent cluster DNS/service networking - Prefect server restarts - lease state not surviving as expected across server issues --- 2. Check worker logs at the same timestamp Look at the Kubernetes worker logs for the run that was cancelled. You want to see whether it received and acted on the cancellation request. If cancellation happened cleanly, you’d expect the worker to attempt to delete the infrastructure job. --- 3. Don’t manually delete K8s jobs unless you want retry behavior If retries are configured on the deployment/flow, manual job deletion can trigger another attempt. Instead, prefer cancelling the flow run from Prefect, so the orchestration and infrastructure cleanup stay in sync. --- 4. Consider reducing or disabling retries while debugging If these cancellations are creating wasteful reruns, temporarily set retries to

0

for this deployment while you debug the underlying lease-loss issue. ---

5. If startup/provisioning is slow, review deployment concurrency configuration If the problem happens mostly during long scheduling/provisioning windows, it can be related to the lease lifecycle around acquiring and renewing slots. If you’re setting deployment concurrency in code, there is a grace-period concept involved in the lease. If you want, I can help you inspect your exact deployment definition and whether this is tunable in your version. --- Practical summary - Expected: losing the deployment concurrency lease causes Prefect to cancel the flow run. - Not ideal / worth investigating: the Kubernetes pod keeps running and still uses RAM. - Also expected-ish: manually deleting the K8s job can cause a retry if retries are configured. --- What I’d do next 1. Correlate timestamps across: - flow logs - worker logs - Prefect server logs - Kubernetes events for that pod/job 2. Check whether the worker actually attempted infra deletion. 3. Confirm whether the affected deployment has retries configured. 4. Verify whether Prefect server had any restart/network blip at the same moment the

410 Gone

happened. --- If you want, send me: - your deployment concurrency config - whether you use a Kubernetes work pool or worker setup - whether retries are configured - relevant worker log lines around the cancellation timestamp and I can help narrow down whether this is expected lease loss from infra/network conditions or a Kubernetes cleanup bug. Relevant resources: - Prefect docs - Workers - Kubernetes deployment patterns

Yes — your theory about upgrading makes sense. If you’re on 3.6.20, and your flow code is CPU-bound or otherwise blocks the event loop/thread that was responsible for lease renewal, that can absolutely explain the lost lease. The change you mentioned in 3.7.x is relevant here. So on the cause of the cancellation: upgrading is a very reasonable mitigation. --- But why doesn’t the pod terminate? This is the important architectural point: The flow runtime cancelling itself is not the same thing as Kubernetes immediately killing the pod. With a Kubernetes work pool, the sequence is roughly: 1. Worker submits the K8s Job 2. Pod starts and runs the flow 3. Inside the pod, Prefect detects lease renewal failure 4. The runtime cancels the flow run from inside the container 5. The worker is supposed to notice that cancellation and then delete the K8s Job / pod So the runtime and the worker are decoupled. That explains your observation: - worker log shows

Creating Kubernetes job...

- later, flow crashes - worker logs show nothing That strongly suggests the worker never observed or processed the later cancellation event, so it never issued the Kubernetes job deletion. --- Why that can happen The cancellation propagation is asynchronous. The pod doesn’t directly tell Kubernetes “kill me now” just because Prefect cancelled the run logically. Instead, Prefect state changes have to be observed by the worker/observer path. So if that path misses or delays the event, you can get: - run marked cancelled/crashed in Prefect - pod still alive in Kubernetes - memory/resources still consumed That’s why the lack of worker logs is meaningful: it suggests the cleanup path was never triggered. --- Why manual K8s Job deletion caused a retry Because from Prefect’s perspective, deleting infra underneath an active/retryable run can look like infrastructure failure, not a graceful orchestrated cancellation. If retries are configured, Prefect may schedule another attempt. So for these cases: - deleting the job is infrastructure failure semantics - cancelling the flow run in Prefect is orchestration semantics Those are not equivalent. --- Short answer to your direct question How come the pod isn’t terminating and freeing resources? Because the lease-loss cancellation happens inside the runtime, but Kubernetes job cleanup depends on the worker noticing that cancellation and calling job deletion. If the worker never receives/processes that cancellation event, the pod can remain orphaned and keep consuming resources. So: - run cancellation is expected - orphaned pod is not ideal and points to missed cleanup/observer behavior --- What I’d recommend 1. Upgrade first If you suspect CPU-bound work is delaying lease renewal, upgrading from 3.6.20 to a newer 3.7.x is a good first move. That may reduce or eliminate the original lease-loss problem. --- 2. Check whether the container process actually exits on its own Even before worker cleanup, an in-process cancellation should often cause the Python process to exit soon after, unless: - the code is stuck in a CPU-bound/native call - it’s blocked in a place where cancellation isn’t observed promptly - subprocesses/child processes keep the container alive - the main process catches/swallows cancellation badly So one question is: Is the pod still actively running your Python process, or is it stuck in termination / hanging due to another process? You can inspect: - pod status - container state - process list inside the pod - final pod logs after the cancellation message --- 3. Inspect worker cancellation/observer health Since the worker never logged cancellation handling, I’d focus there too. Things to check: - does the worker stay healthy the whole time? - any websocket/event-stream disconnects in worker logs? - any observer-related warnings/errors? - if the worker restarts, does cleanup happen later? ---

4. Consider adding Kubernetes TTL / cleanup safeguards This won’t fix root cause, but it can limit resource leakage. For example, if jobs finish/terminate eventually, using a finished-job TTL can help reduce lingering resources. It won’t solve a truly still-running pod, but it helps with completed/failed job cleanup. --- 5. Temporarily disable retries while debugging If retries are making orphan/debug situations worse, set retries to

0

temporarily so manual cleanup doesn’t spawn more attempts. --- What I suspect most in your case Most likely combination: - on 3.6.20, lease renewal gets starved/blocked - runtime self-cancels - worker does not observe/process the cancellation event - pod remains running because no one deletes the K8s Job - manual K8s deletion then triggers retry behavior So I think there are two separate concerns: 1. Why the lease is lost → upgrade likely helps 2. Why infra cleanup didn’t happen → worker cancellation observation/cleanup path --- If you want, I can help you pin this down further. Please send: - your worker version and server version - whether your flow has

retries

configured - whether the pod remains in

Running

or

Terminating

- what

kubectl describe pod ...

shows after the crash - whether the main container process is still alive - any worker log lines containing

cancel

,

observer

,

websocket

, or

poll

And I can help determine whether this is: - expected delayed cancellation behavior, - a stuck Python/container process, - or a worker/observer cleanup bug. Useful docs: - Workers - Kubernetes work pools on Kubernetes - Concurrency limits

Short answer: probably not. A broken

kopf

observer is unlikely to be the direct reason the pod is not getting terminated after this kind of cancellation. Why? Because these are two different mechanisms. --- What

kopf

is for In the Kubernetes worker, the

kopf

-based observer is mainly used to watch Kubernetes objects and reflect what’s happening back into Prefect, for things like: - pod/job lifecycle observation - crash detection - event replication / state updates It is mostly observational. --- What actually kills the pod/job The actual cleanup path for cancellation is separate: - Prefect decides a run should be cancelled - the worker receives/processes that cancellation - the worker calls its Kubernetes cleanup logic - that logic deletes the Kubernetes Job - Kubernetes then terminates the pod(s) So the important actor for termination is the worker’s cancellation cleanup path, not

kopf

itself. --- So can

kopf

be involved indirectly? Indirectly maybe, directly probably no. A broken

kopf

observer could cause problems like: - missing pod/job state updates - delayed or missing crash detection - poor visibility into what happened But it should not be the component responsible for actually issuing the delete for the K8s Job when a run is cancelled. So if the pod remains running, the more likely explanations are: - the worker never received/processed the cancellation cleanup event - the worker lost connectivity / missed the event stream - the cleanup path was interrupted - the job deletion call to Kubernetes never happened - the process inside the pod didn’t exit, and no external deletion occurred --- In your case, the strongest clue is this:

worker logs show only

Creating Kubernetes job...

and nothing when the run crashes

That points more to: - missed cancellation propagation / cleanup handling than to -

kopf

failing to watch pod state If

kopf

were the only problem, I’d expect issues in status/crash observation, but not necessarily failure to terminate a cancelled job. --- What to check next I’d focus on whether the worker’s cancellation/cleanup machinery is healthy, not just `kopf`: - worker connectivity to Prefect API/server - any websocket/event-stream disconnects - worker restarts - whether cancellation polling is happening - whether the Kubernetes Job ever received a delete call - whether the pod process actually exited on its own Also check Kubernetes events on the job/pod:

bash
kubectl describe job <job-name> -n <namespace>
kubectl describe pod <pod-name> -n <namespace>

If there was never any delete/termination attempt, that supports “worker cleanup didn’t fire”. --- Bottom line - Could

kopf

being broken explain missing observability? Yes. - Could it be the main reason cancelled runs are not terminating their pods? Probably no. - More likely: the worker cleanup/cancellation path didn’t run or didn’t finish. If you want, I can help you build a precise diagnostic checklist for this case: - what worker logs to grep - what pod/job fields to inspect - how to tell whether the worker missed the cancellation versus Kubernetes refusing termination.