Pradnyesh Choudhari
11/03/2025, 5:44 PMMarvin
11/03/2025, 5:44 PMMarvin
11/03/2025, 5:54 PM--work-queue express, it will poll only the express queue and will never pull from default, regardless of priority. To poll multiple queues from the same pool, either omit --work-queue to poll all queues in the pool or list both:
prefect worker start -p <your-pool>
# or explicitly:
prefect worker start -p <your-pool> -q express -q default
Other frequent causes to check:
- The default queue is paused
- The default queue has a concurrency limit that is 0 or already saturated
- No workers are subscribed to the default queue (or worker is offline)
- Your deployment isn’t actually assigned to the default queue name you think it is
- Queue has a filter that excludes your runs (e.g., tag filters)
- Runs are not yet eligible (scheduled in the future), or stuck in AwaitingConcurrencySlot because of tag/deployment/global concurrency limits
- The work pool itself is paused
Helpful CLI checks (verified commands)
- See your queues, priorities, paused status, and limits:
prefect work-queue ls -p <your-pool> -v
- Inspect a specific queue:
prefect work-queue inspect default -p <your-pool>
- Preview upcoming runs for a queue:
prefect work-queue preview default -p <your-pool> --hours 1
- Confirm what your worker is polling (check worker logs on start; it prints “Subscribed queues”). If you need to subscribe to both queues:
prefect worker start -p <your-pool> -q express -q default
How priorities are used
- When a worker asks the API for work from a pool, the API selects the next eligible flow run from the highest-priority non-empty queue among the queues that worker is subscribed to.
- If multiple queues share the same priority, ordering falls back to run readiness (e.g., earliest scheduled time).
- Priorities do not preempt already-running work; they affect dispatch order at scheduling time.
If you share:
- The exact command you use to start the worker
- The output of:
prefect work-queue ls -p <your-pool> -v
prefect work-queue inspect default -p <your-pool>
- A flow run id that did not start as expected
…I can pinpoint the root cause quickly and suggest the exact fix.Pradnyesh Choudhari
11/03/2025, 6:40 PMMarvin
11/03/2025, 6:41 PMMarvin
11/03/2025, 6:44 PMprefect worker start -p pr-worker-pool -q express -q default
- On worker startup, check logs for “Subscribed queues: …” and “Capacity …”.
2) Queue concurrency limit reached on default
- default has a concurrency limit of 2. If two runs from default are already RUNNING, new ones won’t start.
- Quick ways to confirm:
prefect work-queue preview default -p pr-worker-pool --hours 1
prefect work-queue read-runs default -p pr-worker-pool
prefect flow-run ls --state RUNNING --limit 50
- As a test, you can temporarily clear the queue limit and see if runs start:
prefect work-queue clear-concurrency-limit default -p pr-worker-pool
(You can set it back after testing.)
3) Deployment/tag/global concurrency limits
- If runs in default are AwaitingConcurrencySlot due to tag/deployment/global limits, they’ll remain queued even though the queue is polled. Check run states in the UI or:
prefect flow-run ls --limit 50
Look for states like AWAITINGCONCURRENCY or SCHEDULED vs RUNNING.
4) Queue filters
- If the default queue has a filter (e.g., tags) that excludes your runs, the worker will poll but won’t get eligible work. You can review/edit queue filters in the UI for that work pool/queue.
5) Work pool state or limits
- Confirm the pool itself is not paused and doesn’t have a pool-level concurrency limit that’s already saturated:
prefect work-pool inspect pr-worker-pool
Next best steps to pinpoint the blocker
Please share:
- How you start your worker (the exact command)
- Output of:
prefect work-pool inspect pr-worker-pool
prefect work-queue read-runs default -p pr-worker-pool
prefect work-queue preview default -p pr-worker-pool --hours 1
prefect flow-run ls --state RUNNING --limit 50
- One flow run ID from default that is not starting
With that, I can tell you precisely whether it’s queue concurrency, worker capacity, a filter/eligibility issue, or a deployment/tag/global limit causing the hold-up.