Short answer: with collision_strategy=CANCEL_NEW, new runs should be cancelled immediately when the single slot is occupied — they should not “queue.” If you’re seeing a pile-up, it’s usually one of these: 1) The deployment didn’t actually pick up your YAML change - You must redeploy after editing prefect.yaml. Run this from your project directory:

prefect deploy

- Verify what the server thinks your deployment config is:

prefect deployment inspect "<flow_name>/<deployment_name>"

You should see something like:

concurrency_limit: { limit: 1, collision_strategy: CANCEL_NEW, ... }

If it shows ENQUEUE or no collision_strategy, the change wasn’t applied. 2) You’re looking at runs waiting for a worker, not a concurrency slot - Deployment concurrency only controls how many runs of that deployment can run at once. - If you do not have a running worker or your work pool/queue is constrained, runs will accumulate in Scheduled/Late states and appear “queued” regardless of deployment concurrency. - Make sure a worker is running for the correct pool/queue:

prefect worker start -p <your-work-pool> -q <your-queue>

- Inspect one of the piled-up runs:

prefect flow-run inspect <flow_run_id>

If the state is Scheduled or Late, that’s worker capacity. If it’s AwaitingConcurrencySlot, that indicates ENQUEUE behavior. 3) The YAML is in the wrong place/format - The concurrency_limit block must live under the specific deployment entry in prefect.yaml, e.g.:

deployments:
  - name: my-deployment
    entrypoint: path/to/flow.py:my_flow
    concurrency_limit:
      limit: 1
      collision_strategy: CANCEL_NEW

If it’s top-level or under work_pool, it will be ignored. 4) Subflow runs - Deployment-level limits apply only to runs of that deployment. Subflows (runs created inside a parent flow) are not governed by the deployment’s concurrency_limit. What you should expect with CANCEL_NEW - While one run is using the single slot, any new run that reaches its start time will immediately transition to Cancelled with a message indicating the deployment concurrency limit was reached. It will not move into AwaitingConcurrencySlot or otherwise queue. If you can share: - The full deployment section of your prefect.yaml - The output of:

prefect deployment inspect "<flow_name>/<deployment_name>"

- The state shown for one of the “queued” runs (Scheduled/Late vs AwaitingConcurrencySlot) …I can pinpoint which of the above is happening. Useful docs: - Deployments and concurrency limits - prefect.yaml reference - States (see AwaitingConcurrencySlot vs Scheduled/Late)

Sorry, I encountered an error while processing your request. Please try again.

If it’s worker capacity, the fix is to make sure a worker is actively polling the same work pool/queue your deployment targets and that the pool/queue aren’t paused or rate-limited. Here’s a quick checklist: 1) Confirm where your deployment is pointing - Check the deployment’s work pool and queue:

prefect deployment inspect "<flow_name>/<deployment_name>"

Note the work_pool name and (optional) work_queue name. 2) Ensure a worker is running for that pool/queue - Start a worker that polls the same pool (and queue, if specified):

prefect worker start -p <your-work-pool> -q <your-work-queue>

- For Docker/Kubernetes workers, run with the extras installed:

uv run --with 'prefect[docker]' prefect worker start -p <docker-pool>
uv run --with 'prefect[kubernetes]' prefect worker start -p <k8s-pool>

3) Check that the pool/queue aren’t paused or limited to 0 - List pools and queues to see status/backlog:

prefect work-pool ls
prefect work-queue ls -p <your-work-pool> --verbose

- Resume if paused:

prefect work-pool resume <your-work-pool>
prefect work-queue resume -p <your-work-pool> <your-work-queue>

- Raise or clear overly strict concurrency limits at the pool/queue level:

prefect work-pool set-concurrency-limit <your-work-pool> <n>
prefect work-queue set-concurrency-limit -p <your-work-pool> <your-work-queue> <n>

4) Scale workers if needed - If the queue has a backlog (many Scheduled/Late runs), start more workers (or increase resources for the existing worker) to drain it faster. Multiple workers can poll the same pool/queue. 5) Verify run states to confirm it’s a worker wait - Inspect a queued run:

prefect flow-run inspect <flow_run_id>

If the state is Scheduled or Late, that indicates worker availability/capacity. If you ever see AwaitingConcurrencySlot, that’s deployment concurrency (ENQUEUE) rather than worker capacity. Notes for upgrading from 2.x - In 3.x, there is no agent. Use workers instead:

prefect worker start -p <your-work-pool> -q <your-work-queue>

If you share: - The work_pool/queue from your deployment inspect output - The output of

prefect work-queue ls -p <your-work-pool> --verbose

…I can help you pinpoint the exact change (start a worker, resume a pool/queue, or adjust limits). Docs for reference: - Work pools overview - Workers overview - Managing work pools (create, pause/resume, limits) - Worker CLI - Work pool CLI - Work queue CLI

Late and Pending point to worker/pool/queue issues, not deployment concurrency. What those states mean - Late: the scheduled start time has passed but no worker has picked up the run. - Pending: the run acquired a slot and is ready, but has not transitioned to Running yet (usually awaiting worker submission/infrastructure start). Most common causes and fixes 1) No worker polling the right pool/queue - Confirm where your deployment is pointed:

prefect deployment inspect "<flow_name>/<deployment_name>"

- Start a worker for that pool (and queue if set):

prefect worker start -p <your-work-pool> -q <your-work-queue>

For Docker/Kubernetes pools, ensure extras are installed:

uv run --with 'prefect[docker]' prefect worker start -p <docker-pool>
uv run --with 'prefect[kubernetes]' prefect worker start -p <k8s-pool>

2) Pool/queue is paused or starved by limits/priority - Check status/backlog:

prefect work-pool inspect <your-work-pool>
prefect work-queue ls -p <your-work-pool> --verbose
prefect work-queue preview -p <your-work-pool> <your-work-queue>

- If paused, resume:

prefect work-pool resume <your-work-pool>
prefect work-queue resume -p <your-work-pool> <your-work-queue>

- If concurrency is 0 or too low, raise it:

prefect work-pool set-concurrency-limit <your-work-pool> <n>
prefect work-queue set-concurrency-limit -p <your-work-pool> <your-work-queue> <n>

- If you have multiple queues, lower-priority queues can lag. Consider dedicating a worker to the affected queue or adjusting priorities. 3) Worker/pool type mismatch or infra can’t start - The worker must match the pool type (process, docker, kubernetes). Check:

prefect work-pool ls

- Review worker logs for errors pulling images, creating K8s jobs, auth issues, etc. - For Docker/K8s, verify base job template, image availability, credentials, and cluster permissions. 4) Time skew - If server and worker clocks are out of sync, runs can appear Late. Ensure NTP/time sync on both. 5) Network/connectivity - Workers must reach your Prefect server/Cloud. Check proxies, firewalls, PREFECT_API_URL, and worker logs for connection errors. Quick “from 2.x agent to 3.x worker” setup - Create a pool and “default” queue, point your deployment to them, and start a worker:

prefect work-pool create "my-pool" --type process
prefect work-queue create -p "my-pool" default
prefect deploy
prefect worker start -p "my-pool" -q default

How to confirm what’s blocking a specific run - Inspect a run to see exact state and message:

prefect flow-run inspect <flow_run_id>

- Scheduled/Late → no worker or capacity/priority/paused queue - Pending → usually waiting on worker submission or infra start (check worker logs) - AwaitingConcurrencySlot → deployment-level concurrency with ENQUEUE (not worker) If you paste: -

prefect deployment inspect "<flow>/<deployment>"

-

prefect work-queue ls -p <pool> --verbose

- State + message from

prefect flow-run inspect <id>

…I’ll pinpoint the exact cause. References - Work pools overview - Workers overview - States - Managing work pools (pause/resume, limits) - Worker CLI - Work queue CLI

Got it — even with 2 workers online, flows can sit in Late or Pending if those workers aren’t polling the exact pool/queue you’re using, are constrained by pool/queue limits/priority, or are failing to submit jobs. Here’s a focused checklist to pinpoint and fix it: 1) Confirm routing: pool and queue - See where your deployment is pointed:

prefect deployment inspect "<flow_name>/<deployment_name>"

Note the work_pool and work_queue. Your workers must poll these. - If you launched workers with -q, they will only pull from those queues. Make sure they match the deployment’s queue. To be safe, start a dedicated worker for that queue:

prefect worker start -p <your-work-pool> -q <your-work-queue>

2) Check pool/queue status and capacity - Look for paused/limited queues or starvation due to priority:

prefect work-queue ls -p <your-work-pool> --verbose
prefect work-queue preview -p <your-work-pool> <your-work-queue>
prefect work-pool inspect <your-work-pool>

- If paused, resume:

prefect work-pool resume <your-work-pool>
prefect work-queue resume -p <your-work-pool> <your-work-queue>

- If limits are too strict (0 or very low), raise them:

prefect work-pool set-concurrency-limit <your-work-pool> <n>
prefect work-queue set-concurrency-limit -p <your-work-pool> <your-work-queue> <n>

Tip: If you have multiple queues with priorities, a high-priority queue can starve a lower-priority queue. Either dedicate one worker to the affected queue or adjust priorities. 3) Verify worker type and logs - The worker type must match the pool type (process/docker/kubernetes). - Check worker startup logs — they print the pool and queues being polled. If you don’t see your target queue listed, that worker will never pick those runs. - If runs are stuck in Pending, look for submission/infrastructure errors in worker logs. You can increase verbosity:

PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p <your-work-pool> -q <your-work-queue>

4) Inspect a stuck run to confirm the blocker

prefect flow-run inspect <flow_run_id>

- Late: scheduled time passed, no worker picked it up → routing/capacity/priority/paused queue. - Pending: slot acquired but not Running → worker isn’t submitting or infra can’t start (check worker type/config and logs). If you paste: - The work_pool and work_queue from

prefect deployment inspect "<flow>/<deployment>"

- Output of

prefect work-queue ls -p <pool> --verbose

and

prefect work-queue preview -p <pool> <queue>

- How you started each worker (the exact command or the startup log line showing pool/queues) …I’ll identify exactly why those runs are Late/Pending and what to change. Helpful docs: - Work pools overview - Workers overview - Work queue CLI - Manage pools (pause/resume, limits, queues)

Thanks — that pattern (Late backlog, then Pending but never Running) almost always means “a worker claimed the run but could not submit/start infrastructure.” The fix depends on your worker type and configuration. Here’s a tight checklist to find and resolve the blocker: 1) Inspect a stuck run for the exact reason - Look at the state message/details:

prefect flow-run inspect <flow_run_id>

You should see messages like “Claimed by worker …”, “Submitting…”, or an error from the worker/infrastructure. Share this message if you can. 2) Check worker logs right after a claim - Start (or restart) the worker with debug logs to capture submission errors:

PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p <your-work-pool> -q <your-work-queue>

- In healthy runs you’ll see roughly: “Claimed flow run” → “Submitting” → “Created job” → “InfrastructurePending/Running”. If it stops after “Claimed” or shows exceptions, that’s your root cause. 3) Verify worker type matches pool type - Process pool needs a process worker; Docker pool needs a docker worker; K8s pool needs a kubernetes worker. Mismatches won’t be able to submit jobs. - Check:

prefect work-pool inspect <your-work-pool>

If the type differs from the worker you started, start the correct worker. 4) If you’re using a Process worker - Ensure the flow code is available on the worker machine. In Prefect 3.x, use source-based deployments or install your project on the worker. - Typical failure: worker can’t import the flow module (no project files in that environment), so it never reaches Running. Fix by: - Installing your package on the worker host (venv/conda), or - Using source-based deployments (e.g., Git repo) so the worker can fetch code. 5) If you’re using a Docker worker - Validate the image actually exists and is pullable from the worker host:

docker pull <your-image>

- If it’s private, confirm registry credentials are in the base job template or on the host. - Make sure the image contains your flow code and a compatible Python/Prefect version. 6) If you’re using a Kubernetes worker - Confirm permissions and namespace:

kubectl auth can-i create jobs --namespace <ns>

- Check events for failed Job/Pod creation and image pull errors:

kubectl get jobs,pods -n <ns>
kubectl describe job/<name> -n <ns>
kubectl describe pod/<pod> -n <ns>

- Verify the base job template (env vars, service account, image, resources). 7) Sanity checks - Worker actually polling the right queue(s). At startup, the worker logs the pool and queues; confirm your deployment’s queue is listed. - Time sync on server/worker (NTP) to avoid Late artifacts. - If Pending eventually flips to Cancelled with a message about concurrency slot lease, increase

grace_period_seconds

in your deployment’s

concurrency_limit

to give infra more time to start. Fastest way to isolate the problem - Kick off a run and watch it while tailing worker logs:

prefect deployment run "<flow_name>/<deployment_name>" --watch

If the run stays in Pending and the worker logs show submission errors (image not found, import error, k8s forbidden, etc.), fix that error and re-run. If you share: - Worker type and exact start command(s) -

prefect deployment inspect "<flow>/<deployment>"

(to confirm pool/queue) - The state message from

prefect flow-run inspect <id>

- A few lines of worker logs after “Claimed flow run …” …I can pinpoint the exact fix. Refs: - Workers overview - Work pools overview - Managing pools/queues and job templates