Hey Prefect team, we've had a case where our flows hit a bug causing them to hang. It turns out that just canceling them doesn't kill the underlying infrastructure (k8s) causing a lot of zombie jobs taking up resources (filed issue here). It looks like the current worker implementation explicitly only cancels PENDING runs and not RUNNING or CANCELLING. What's the reason behind this and can we change it to kill running (stuck) flows too?

Hey @Chu! Normally, the cancellation of the flow run and teardown of the infrastructure are handled by a process running within the infrastructure. If your flow runs are hanging and cancellation isn't working, it possible that hang is affecting the cancellation process too. We plan to improve this cleanup by introducing a separate channel to notify workers to clean up infrastructure when flow runs get stuck in states like

CANCELLING

or

PENDING

.

cancellation of the flow run and teardown of the infrastructure are handled by a process running within the infrastructure

In a context where one is using k8s as the underlying , for example, this would be the prefect k8s worker?

No, not if you're running workflows on K8s. In that case, the process that starts up with

prefect flow-run execute

inside the K8s pod starts the flow run in a subprocess and opens a websocket to listen for cancellation state changes. If a canceling state change is received, then the parent process terminates the flow run execution process and exits. If the parent process hangs or gets suspended via a

SIGSTOP

, but doesn't exit, then you can get zombie infra. Those are the cases that we aim to solve with this additional communication channel. Do you have any theories why job processes might be hanging?

Ah, yep - makes sense We're trying to understand the job hang - one part of it is https://github.com/PrefectHQ/prefect/pull/21612; because of this issue the flow run hangs forever without making progress, which I think then explains the zombie infra issue There's a chance by merging the above we also fix this problem in a narrow sense (though agree that there should be a path outside of the cancel-in-infra path for this exact reason - seems like fundamentally a Worker responsibility to me)

Hey @alex, thanks for the response - that makes sense. Is there a way to "force cancel" a flow then? It seems like something that should be supported without having to wait for the prefect process to respond.

Unfortunately, there isn't a way to do that via Prefect today. The best workaround is to delete the K8s job directly, but I know that isn't a great solution. I'll prioritize the solution to make cancellation reliable even when flow runs hang because I've seen others struggling with this as well.

@Chu For k8's OSS, I responded to @alex in another thread about what I found worked https://prefect-community.slack.com/archives/CL09KU1K7/p1776705073185339?thread_ts=1776223842.260949&cid=CL09KU1K7