<@ULVA73B9P>, I have two Prefect deployments that ...
# marvin-aii
Ilya Yakavets
12/18/2025, 5:07 PM@Marvin, I have two Prefect deployments that both use a global concurrency limit of 1 (so only one run can execute at a time). Each run typically takes ~5 minutes, and each deployment is scheduled on a 10‑minute interval.
What I’m seeing: deployment A keeps running normally, but deployment B ran once and then subsequent runs stay stuck in “Awaiting concurrency slot” and never seem to execute.
• When a run is scheduled but can’t start due to concurrency, does Prefect apply any “stale/late” cutoff or timeout that eventually prevents the run from starting?
• Is there a configurable setting like max lateness, schedule catch-up, deadline, max number of late scheduled runs, or similar that controls whether an “awaiting slot” run is still eligible to start later?
• If the correct behavior is “it should run once a slot frees”, what are the common causes of it never starting (e.g., stuck RUNNING run holding the slot, work pool/worker concurrency vs deployment concurrency, queue/backlog behavior), and what should I check in the UI/API to confirm?
m
Marvin
12/18/2025, 5:07 PM✅ thought for 303.6 seconds
Marvin
12/18/2025, 5:12 PMThanks for the detailed context — that helps. Assuming Prefect 3.x:
Short answers
- Does “awaiting slot” ever time out? No. Prefect does not apply a hard “lateness cutoff” that stops a run from starting later. A run waiting for a concurrency slot remains eligible and will keep retrying.
- Is there a setting like max lateness/catch‑up/deadline/max late runs that prevents starts? No. There’s no built‑in deadline or max backlog for concurrency‑gated runs. You can reduce backlog by using
collision_strategy=CANCEL_NEWPREFECT_SERVER_DEPLOYMENTS_CONCURRENCY_SLOT_WAIT_SECONDSPREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATIONconcurrency_options.grace_period_secondsPREFECT_SERVER_SERVICES_LATE_RUNS_AFTER_SECONDScollision_strategy=CANCEL_NEWconcurrency_limitconcurrency_options.collision_strategygrace_period_secondsMarvin
12/18/2025, 5:12 PMcurl -X POST "$PREFECT_API_URL/api/flow_runs/filter" \
-H "Content-Type: application/json" \
-d '{
"flow_runs": {
"deployment_id": {"any_": ["<deployment_id>"]},
"state": {"type": {"any_": ["PENDING", "RUNNING"]}}
}
}'
```
- Work pool/queue limits and priority
- Check the work pool’s and queue’s concurrency limits and queue priority. If A’s queue has higher priority and the pool limit is tight, B can be starved.
- Verify a healthy worker is attached to B’s queue.
- Docs: Work pools and queues
- Late runs and SLAs (for observability)
- If runs are marked Late, that’s expected during concurrency waits; it doesn’t block starting.
- Docs: States and SLAs
Why you might see A running regularly and B “never” starting
- If A’s schedule creates a steady backlog (or A’s queue has higher priority), the worker may continually service A’s runs. Since the global limit is 1, B keeps retrying acquisition but loses the race most of the time.
- If A and B are scheduled simultaneously, one may consistently beat the other to the slot. A small schedule offset (e.g., start B 5 minutes after A) often restores fairness.
- If there’s a stuck run (RUNNING or long‑PENDING) holding the slot, B won’t get a turn until it’s released — check A’s and B’s run history and the concurrency page for the holder.
Practical next steps
- Verify both deployments reference the same global concurrency key and confirm which runs are currently holding it (Concurrency page).
- Check queue/pool concurrency and queue priority. Ensure B’s queue has a worker and is not deprioritized.
- Reduce starvation by offsetting schedules (e.g., A at 00/10/:20, B at 05/15/:25).
- If you want to prevent backlog growth instead of waiting, set
collision_strategy=CANCEL_NEWgrace_period_secondsi
Ilya Yakavets
12/18/2025, 8:46 PM@Marvin @Marvin, can you confirm the correct behavior of Prefect 3.x when multiple deployments share a global concurrency limit=1: does the scheduler guarantee fairness between deployments, or can one deployment starve another indefinitely if their runs are scheduled at the same time? If starvation is possible, what’s the recommended mitigation—schedule offset via anchor_date, or a supported fairness/priority mechanism?
If you want, I can also patch the dashboard “Cancel all” to use Prefect’s real /set_state cancellation so it actually works on your server.
m
Marvin
12/18/2025, 8:46 PM✅ thought for 160.6 seconds
Marvin
12/18/2025, 8:49 PMGreat question — and yes, this can definitely bite you.
Direct answer
- Prefect 3.x does not guarantee fairness or round‑robin between deployments that share a global concurrency limit (e.g., limit=1). If their runs contend at the same time, one deployment can effectively starve the other.
- Workers pull runs in priority order (by work queue priority) then by earliest scheduled time. There’s no fairness queue per concurrency limit; it’s first-come-first-served at the instant a slot frees.
- So if A routinely grabs the slot first (or has higher queue priority), B can sit in “Awaiting concurrency slot” for a long time.
Recommended mitigations
- Easiest: offset the schedules so they don’t collide. Stagger B’s schedule a few minutes off of A (anchor_date/offset) so their next_scheduled_start_time values aren’t identical.
- Use work queue priorities: put A and B on separate queues and assign priorities intentionally if you want a deterministic winner. Lower number = higher priority.
- Separate queues per deployment: this makes priority explicit and lets you reason about pull order. Keep them in the same work pool if you want shared worker capacity.
- If you want to prevent backlog rather than wait, set deployment
collision_strategy=CANCEL_NEW7 Views