https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have had this problem for a while w...
# ask-marvin
l
@Marvin I have had this problem for a while when flow runs get stuck in a pending state without any reason, or worse, don't transition out of a "running" state at the end of the run. Those block my other runs in case I have a concurrency of 1, that I use quite often to make sure flow runs don't process the same data twice. I'm using prefect 3.4.4. Do you have any idea what is happening here? I cannot find anything on the server or worker logs.
m
✅ thought for 586.2 seconds
Thanks for reporting this — stuck Pending/Running flow runs that block a concurrency=1 setup are definitely frustrating. A few things can cause this in Prefect 3.x, including some edge cases in 3.4.x. Let’s quickly triage and then I’ll suggest mitigations. Quick checks to run - Is your work queue ready and being polled? -
Copy code
prefect work-queue ls
-
Copy code
prefect work-queue inspect <queue_name>
- In the UI, confirm the queue is READY (not PAUSED/NOT_READY) and that at least one worker is polling. - Docs: Work pools and queues - What does Prefect think about the stuck runs? -
Copy code
prefect flow-run ls --state Pending --limit 20
-
Copy code
prefect flow-run ls --state Running --limit 20
-
Copy code
prefect flow-run inspect <run_id>
(look at the current state’s type and “reason” field in the details) - Docs: States and their meanings - Which concurrency mechanism are you using for “concurrency of 1”? - Work queue concurrency (limits concurrent flow runs from that queue) - Deployment-level concurrency/collision strategy (limits concurrent runs for a deployment) - Tag-based/global concurrency (limits tasks across flows) - If you’re using tag-based/global concurrency and a slot is leaked, you can inspect/reset: -
Copy code
prefect concurrency-limit ls
-
Copy code
prefect concurrency-limit inspect <tag>
(shows 
active_slots
) -
Copy code
prefect concurrency-limit reset <tag>
- Docs: Tag-based concurrency and Global concurrency limits Common root causes we see 1) Queue/worker not polling → Pending(NotReady) - If no worker has polled the queue recently or the queue is paused, runs remain Pending with “NotReady”. Check queue status and that your worker is running and connected to the correct pool/queue. - Health ref: Worker health checks 2) Deployment/work-queue concurrency=1 + a run that never finalized - If a run remains Running and never transitions (e.g., infra crash, DB timeout), it will block further runs behind the concurrency limit. You can temporarily unblock by canceling the stuck run: -
Copy code
prefect flow-run cancel <run_id>
- Consider using a deployment-level collision strategy like SKIP so new runs don’t pile up behind a stuck run. 3) Server/database write timeouts under load → flow shows “Finished in state Running()” - We’ve seen reports where, under DB load, the final state write times out so the server never accepts the Completed state; the run appears stuck Running. See: - flow run finishing with state running - In server logs, this looks like “Encountered error during state validation” with SQL timeout traces. - If you see similar errors in API/server logs, upgrading to the latest 3.5.x is recommended as there have been reliability fixes around state transitions and logging. Release notes: Prefect releases. 4) Infra/worker exits before final state/log flush - If the process is OOM-killed, preempted, or otherwise exits before proposing the final Completed state, the server never sees it. This can also explain “no logs” at the end. - Action: check container/job exit codes and cluster/node events; run the worker with debug logs and verify the engine logs reach the “Finished in state Completed()” line. Fast remediation steps to unblock now - Cancel or delete the stuck runs so concurrency can free up: -
Copy code
prefect flow-run cancel <run_id>
- If you use tag/global concurrency, release stuck slots: -
Copy code
prefect concurrency-limit ls
-
Copy code
prefect concurrency-limit inspect <tag>
-
Copy code
prefect concurrency-limit reset <tag>
Helpful details to share so I can pinpoint the cause - Are you on Prefect Cloud or OSS Server? Which server/API version and backing DB? - Worker type and versions (e.g., prefect-kubernetes/prefect-docker) and how you start it (
prefect worker start
command). - Where is “concurrency=1” configured (work queue, deployment, or concurrency limits)? - One or two example flow run IDs that got stuck, plus the output of: -
Copy code
prefect flow-run inspect <run_id>
- Any server logs near the end of those runs (especially errors like “Encountered error during state validation” / SQL timeouts). Related references - States: States - Work pools/queues: Work pools - Worker health: Worker health checks - Concurrency (tag/global): - Tag-based - Global - Known issue under load: flow run finishing with state running - Recent bug report re: tasks stuck Running: tasks stuck in Running If you can share the details above, I’ll help you narrow it down and propose a concrete fix. In parallel, I’d recommend upgrading beyond 3.4.4 to the latest 3.5.x to pick up recent stability fixes around orchestration and logging.
l
@Marvin Here are the behaviors I see on my cluster. The flow run is scheduled, and transitions into a pending state after the job is created. Somehow the job does not seem to get out of the pending state? But I may be misunderstanding. Here are the logs I see on my kubernetes events exporter.
Copy code
2025-11-05 12:29:13.444	
2025-11-05T17:29:13Z DBG Received event involvedObject=<REDACTED> msg="Job has reached the specified backoff limit" namespace=xxx reason=BackoffLimitExceeded
2025-11-05 12:27:45.669	
2025-11-05T17:27:45Z DBG Received event involvedObject=<REDACTED> msg="Started container prefect-job" namespace=xxx reason=Started
2025-11-05 12:27:45.669	
2025-11-05T17:27:45Z DBG Received event involvedObject=<REDACTED> msg="Created container: prefect-job" namespace=xxx reason=Created
2025-11-05 12:27:45.668	
2025-11-05T17:27:45Z DBG Received event involvedObject=<REDACTED> msg="Successfully pulled image xxx\" in 14.744s (14.744s including waiting). Image size: 929738887 bytes." namespace=xxx reason=Pulled



2025-11-05 12:27:31.267	
2025-11-05T17:27:31Z DBG Received event involvedObject=<REDACTED> msg="Pulling image \"xxx\"" namespace=xxx reason=Pulling
2025-11-05 12:27:29.268	
{"metadata":{"name":"<REDACTED>","namespace":"xxx","uid":"xxx","resourceVersion":"<REDACTED>","creationTimestamp":"2025-11-05T17:27:25Z"},"reason":"FailedMount","message":"MountVolume.SetUp failed for volume \"secrets-store-inline\" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers","source":{"component":"kubelet","host":"<REDACTED>"},"firstTimestamp":"2025-11-05T17:27:25Z","lastTimestamp":"2025-11-05T17:27:25Z","count":1,"type":"Warning","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"<REDACTED>","clusterName":"","involvedObject":{"kind":"Pod","namespace":"xxx","name":"<REDACTED>","uid":"<REDACTED>","apiVersion":"v1","resourceVersion":"<REDACTED>","labels":{"batch.kubernetes.io/controller-uid":"<REDACTED>","batch.kubernetes.io/job-name":"<REDACTED>","controller-uid":"<REDACTED>","job-name":"<REDACTED>","job-type":"data-pipeline","prefect.io/deployment-id":"<REDACTED>","prefect.io/deployment-name":"<REDACTED>","prefect.io/deployment-updated":"2025-11-05t17-27-25.131908-00-00","prefect.io/flow-id":"<REDACTED>","prefect.io/flow-name":"<REDACTED>","prefect.io/flow-run-id":"<REDACTED>","prefect.io/flow-run-name":"<REDACTED>","prefect.io/version":"3.4.4","prefect.io/work-pool-id":"<REDACTED>","prefect.io/work-pool-name":"default","prefect.io/worker-name":"<REDACTED>","team":"data-platform"},"annotations":{"karpenter.sh/do-not-disrupt":"true"}}
2025-11-05 12:27:29.268	
2025-11-05T17:27:29Z DBG Received event involvedObject=<REDACTED> msg="MountVolume.SetUp failed for volume \"secrets-store-inline\" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers" namespace=xxx reason=FailedMount
2025-11-05 12:27:27.673	
2025-11-05T17:27:27Z DBG Received event involvedObject=<REDACTED> msg="Successfully assigned xxx/<REDACTED> to <REDACTED>" namespace=xxx reason=Scheduled
2025-11-05 12:27:27.269	
2025-11-05T17:27:27Z DBG Received event involvedObject=<REDACTED> msg="Created pod: <REDACTED>" namespace=xxx reason=SuccessfulCreate
m
✅ thought for 69.8 seconds
Thanks — these events are very helpful. They point to two Kubernetes-level issues that explain why your Prefect flow run never leaves Pending: What your events show - FailedMount: -
Copy code
MountVolume.SetUp failed for volume "secrets-store-inline": ... driver name <http://secrets-store.csi.k8s.io|secrets-store.csi.k8s.io> not found in the list of registered CSI drivers
- This means the pod tried to mount a Secrets Store CSI volume but the CSI driver isn’t installed on that node (or cluster). When a required volume can’t mount, the pod can’t start the flow process, so Prefect never gets to propose the Running/Completed states. - BackoffLimitExceeded: -
Copy code
Job has reached the specified backoff limit
- After scheduling, the container was Created/Started, then crashed repeatedly until the Job hit its backoff limit. That typically happens when the container command exits quickly with a non-zero status (wrong entrypoint, missing dependencies, bad env, etc.). Because the flow process never actually starts, Prefect still shows the run as Pending. Why this leads to “stuck Pending” In Kubernetes-based execution, a flow run only transitions to Running once the flow process in the pod starts and the engine proposes the Running state. If the pod can’t mount required volumes or the container crashes immediately, the engine never runs, so Prefect leaves the run in Pending. With a concurrency limit of 1, that Pending run will block subsequent runs. What to do next (concrete steps) - Fix the Secrets Store CSI dependency or scheduling: - Install the Secrets Store CSI driver cluster-wide, or - Remove the 
secrets-store-inline
volume from the job template if you don’t need it, or - Restrict these jobs to nodes that have the CSI driver (nodeSelector/taints+tolerations). - Find the actual container error that caused BackoffLimitExceeded: -
Copy code
kubectl describe job <job_name> -n xxx
  kubectl get pods -l job-name=<job_name> -n xxx
  kubectl describe pod <pod_name> -n xxx
  kubectl logs <pod_name> -n xxx -c prefect-job --previous
- Look for the container’s exit code and any Python/command errors. Common causes: - Prefect not installed in the image (so the entrypoint 
python -m prefect.engine
fails) - Wrong working directory or missing flow code - Missing 
PREFECT_API_URL
/ 
PREFECT_API_KEY
in the pod env - Image entrypoint overrides Prefect’s command - Make sure your Kubernetes worker is watching and surfacing infra failures: - Start the worker with debug logs: -
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p <pool_name>
- In your Kubernetes work pool base job template, consider enabling log streaming so you get pod logs in Prefect: - Set the Kubernetes job template (prefect-kubernetes) with 
stream_output: true
. - Unblock your queue for now (so concurrency=1 doesn’t jam): - Cancel stuck runs: -
Copy code
prefect flow-run ls --state Pending
      prefect flow-run cancel <run_id>
- If you use any tag/global concurrency limits, clear leaked slots: -
Copy code
prefect concurrency-limit ls
      prefect concurrency-limit inspect <tag>
      prefect concurrency-limit reset <tag>
- Consider upgrading - There have been reliability fixes around state updates and infra handling since 3.4.4. I recommend upgrading Prefect and the kubernetes integration to the latest 3.5.x line. - Known related issue for context (DB timeouts causing final state issues — not your exact symptom, but relevant to “stuck running/pending” under failures): flow run finishing with state running If you can share: - The Job/Pod describe output and logs for a failing run - Your image’s Prefect version (
pip show prefect
inside the image) and 
prefect-kubernetes
version - The worker start command and the work pool type (Kubernetes) - Where your “concurrency=1” is configured (work-queue vs deployment vs concurrency limit)
…I’ll help pinpoint the precise failure (mount vs entrypoint vs env) and propose a concrete fix to prevent these from reoccurring.
l
@Marvin and any idea with those errors?
Copy code
2025-11-05 02:30:23.366	
2025-11-05T07:30:23Z DBG Received event involvedObject=xxx msg="Stopping container prefect-job" namespace=xxx reason=Killing
2025-11-05 02:30:22.146	
2025-11-05T07:30:22Z WRN Event discarded as being older then maxEventAgeSeconds event age=5.146737292s event name=xxx event namespace=xxx
m
✅ thought for 298.9 seconds
Good question — those two messages are Kubernetes-level signals and they fit with the earlier failures you shared. What they mean - reason=Killing (Stopping container prefect-job) - Kubelet intentionally terminated the container. This is a generic event that happens for many reasons, including: - Pod deletion/cancellation (user cancel, job controller cleanup after BackoffLimitExceeded) - Container restart due to crash/liveness probe failure (CrashLoopBackOff cycle) - Eviction/preemption or node drain - activeDeadlineSeconds exceeded - By itself, “Killing” doesn’t tell you why — you need the container’s termination reason/exit code. - Event discarded as being older then maxEventAgeSeconds - That’s from your events exporter, not the cluster. It means the exporter dropped an event because it was older than its configured retention window (or there was clock skew). It’s not an error with your job. How to pinpoint the actual cause behind “Killing” Run these against the failing Job/Pod: - Describe the pod and check container termination details: -
Copy code
kubectl describe pod <pod_name> -n xxx
    kubectl get pod <pod_name> -n xxx -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{" "}{.status.containerStatuses[0].lastState.terminated.exitCode}{" "}{.status.containerStatuses[0].lastState.terminated.message}{"\n"}'
- Common outputs: - OOMKilled → container ran out of memory; increase limits/requests or reduce memory usage - Error + exitCode=1 → entrypoint/command failed (missing deps, wrong working dir, missing PREFECT_API_URL/API_KEY, etc.) - DeadlineExceeded → activeDeadlineSeconds killed the pod - Terminated with signal 15 → graceful SIGTERM (pod deletion/cancel/eviction) - Get container logs, including the previous instance if it restarted: -
Copy code
kubectl logs <pod_name> -n xxx -c prefect-job --previous
- List recent events to correlate: -
Copy code
kubectl get events -n xxx --sort-by=.lastTimestamp | tail -n 50
- Check node pressures/preemption if suspected: -
Copy code
kubectl describe node <node_name>
Why this ties to the Prefect behavior you’re seeing - If Kubernetes killed the container before the Prefect engine started (mount failure, immediate crash), the flow run never proposes Running → Prefect leaves it Pending, and with concurrency=1 it will block others. - If the job hit BackoffLimitExceeded, the controller gave up after retries; you’ll see “Killing” during cleanup and the job fails. The Prefect Kubernetes worker should eventually mark the run as Crashed/Failed, but if the container never got far enough to start the engine, the run can sit Pending. Recommendations - Fix the underlying infra issue first (from the termination reason/logs): e.g., install the Secrets Store CSI driver or remove that volume, fix entrypoint/env, add memory/CPU resources, or adjust activeDeadlineSeconds. - Unblock your queue now: -
Copy code
prefect flow-run ls --state Pending
    prefect flow-run cancel <run_id>
- Make infra failures more visible in Prefect: - Run the worker with debug logs: -
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p <pool_name>
- If you’re using the Kubernetes work pool, enable/verify log streaming so pod logs appear on the flow run. - Optional: tune the Job spec in your base job template - backoffLimit (lower it for quicker fail), activeDeadlineSeconds (hard cap), ttlSecondsAfterFinished (auto-cleanup) If you can share the 
kubectl describe pod
termination reason/exit code plus the container logs for one failing run, I can help pinpoint the exact cause and the precise fix. For reference: Prefect on Kubernetes and Workers overview.
v
@Marvin hello. a deployments always need a work-pool?
m
✅ thought for 147.1 seconds
Short answer: for a deployment to actually run, it needs a work pool (and a running worker pulling from that pool). Details: - You can technically create a deployment without assigning a work pool, but any runs (manual or scheduled) will stay pending until you assign it to a pool that has a worker. - If you want to run flows without work pools/workers, don’t use a deployment—run the script directly or use 
flow.serve()
to serve it from the current process. If you’d like, I can share quick steps to create a work pool and start a worker, or show how to use 
flow.serve()
. Are you on Prefect 3.x?
41 Views
Open in Slack
PreviousNext
Powered by