Does anyone else have this issue in Prefect 3? > Since Prefect's

timeout_seconds

is cooperative - it relies on Python's async machinery to raise a

TimeoutError

. If the underlying process is hung (e.g., a stuck CUDA call, a blocked C extension, a deadlocked subprocess), Python can't interrupt it and the timeout never fires. > > If I define an

activeDeadlineSeconds

parameter on the k8's job, Prefect's Kopf observer crashes trying to report the crash and it get's ghost requeued. Would this means that k8's native automatic force kill on a pod or job that get's orphaned or runs to long does not work because Prefect will automatically replace it. Even cancelling the job often does not work because there are blocking processes inside the pod that prevent Prefect's termination signal... Although Prefect 2 had nuances, it did not recreate a pod that's killed by k8's if the job was configured correctly.

prefect==3.6.22
prefect-aws==0.7.6
prefect-docker==0.7.1
prefect-kubernetes==0.7.7
prefect-ray==0.4.5

Even after trying some solutions here: https://prefect-community.slack.com/archives/C04DZJC94DC/p1776218211252279

Hey @Mitch, the pod get recreated based on the backoff limit configured on the K8s job for the work pool. We use the default, which is 6, so the pod would get recreated 6 times. You should be able to short-circuit that by deleting the job directly. That's an ugly solution, and I noticed that these folks are having a similar issue. We have a plan to improve cancellation to work in cases where execution hangs that will solve the issue you're seeing with cancellation here.

@alex thanks for your response, but the solution was actually a combination of things because our back off limit is and always has been 0. The solution was actually a combination of things: • To address the issue of the flow restarting we had to add the environment variable

PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR = "crash"

to the job template so that Prefect will mark it as crashed instead of implicitly retrying • For the cancellation issue, we had to add a redis queue so that such events can be properly handled across server replicas and 1,000 flows

That's good to hear! Are there any other active issues that you're experiencing around zombie infra?

No more zombie infra, just scalability is definitely a pain point on Prefect's end, specifically large concurrency experienced like running 1k flows concurrently while running 50-100k flows daily. There seems to be an extremely large load on the database despite having implemented an RDS Proxy for connection pooling, having a large RDS instance and limiting the workpool connections of the server. We want to scale up to 2k or more concurrent inferencing flows, each running for ~20 minutes, with ~50-100k total but there are transient failures at scale as we try to manage job submission ensure retryability and no wasted compute.

Gotcha, if you share some of the DB operations that are consuming the most CPU then we might be able to find some indexes to unlock some more scale.

I think I have too many workpools and by extension workers, are prefect 2 and prefect 3, so I will start with limiting those as they seem to hammer the same rows of the same db tables