<@ULVA73B9P> help me understand how queues work in...
# ask-marvin
p
@Marvin help me understand how queues work in prefect in a single workpool with different priorities and why even if my express queue with priority 2 is empty why is my default queue with priority 1 not running
m
thought for 601.5 seconds
Short answer first: - In Prefect 3.x, work-queue priority is a positive integer where smaller numbers are higher priority. Priority 1 > 2. - A worker only pulls from the queues it is subscribed to. Priorities only affect ordering among the queues the worker is polling. - If your higher-priority queue is empty, workers can run from lower-priority queues immediately — there’s no backoff — but only if the worker is actually polling those queues and there are eligible runs. Why your default queue (priority 1) isn’t running even though “express” (priority 2) is empty The most common reason is worker subscription: - If you start a worker with
--work-queue express
, it will poll only the
express
queue and will never pull from
default
, regardless of priority. To poll multiple queues from the same pool, either omit
--work-queue
to poll all queues in the pool or list both:
Copy code
prefect worker start -p <your-pool>
# or explicitly:
prefect worker start -p <your-pool> -q express -q default
Other frequent causes to check: - The default queue is paused - The default queue has a concurrency limit that is 0 or already saturated - No workers are subscribed to the default queue (or worker is offline) - Your deployment isn’t actually assigned to the default queue name you think it is - Queue has a filter that excludes your runs (e.g., tag filters) - Runs are not yet eligible (scheduled in the future), or stuck in AwaitingConcurrencySlot because of tag/deployment/global concurrency limits - The work pool itself is paused Helpful CLI checks (verified commands) - See your queues, priorities, paused status, and limits:
Copy code
prefect work-queue ls -p <your-pool> -v
- Inspect a specific queue:
Copy code
prefect work-queue inspect default -p <your-pool>
- Preview upcoming runs for a queue:
Copy code
prefect work-queue preview default -p <your-pool> --hours 1
- Confirm what your worker is polling (check worker logs on start; it prints “Subscribed queues”). If you need to subscribe to both queues:
Copy code
prefect worker start -p <your-pool> -q express -q default
How priorities are used - When a worker asks the API for work from a pool, the API selects the next eligible flow run from the highest-priority non-empty queue among the queues that worker is subscribed to. - If multiple queues share the same priority, ordering falls back to run readiness (e.g., earliest scheduled time). - Priorities do not preempt already-running work; they affect dispatch order at scheduling time. If you share: - The exact command you use to start the worker - The output of:
Copy code
prefect work-queue ls -p <your-pool> -v
prefect work-queue inspect default -p <your-pool>
- A flow run id that did not start as expected …I can pinpoint the root cause quickly and suggest the exact fix.
p
prefect work-queue ls -p pr-worker-pool -v Work Queues in Work Pool 'pr-worker-pool' ┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Name ┃ Priority ┃ Concurrency Limit ┃ Description ┃ ┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ express │ 1 │ 6 │ │ │ default │ 2 │ 2 │ The work pool's default queue. │ └─────────┴──────────┴───────────────────┴────────────────────────────────┘ (**) denotes a paused queue prefect work-queue inspect default -p pr-worker-pool WorkQueue( id='e83dd05f-5d3c-4a8b-97d0-dbca496e6133', created=DateTime(2025, 6, 6, 1, 50, 12, 836184, tzinfo=Timezone('UTC')), updated=DateTime(2025, 11, 3, 18, 40, 17, 687520, tzinfo=Timezone('UTC')), name='default', description="The work pool's default queue.", concurrency_limit=2, priority=2, work_pool_name='pr-worker-pool', work_pool_id=UUID('34b04743-a791-4032-9a15-a899c775489c'), last_polled=DateTime(2025, 11, 3, 18, 40, 17, 681001, tzinfo=Timezone('UTC')), status=WorkQueueStatus.READY ) WorkQueueStatusDetail( healthy=False, late_runs_count=1, last_polled=DateTime(2025, 11, 3, 18, 40, 17, 681001, tzinfo=Timezone('UTC')), health_check_policy=WorkQueueHealthPolicy() ) @Marvin the output you requested
m
thought for 191.6 seconds
Thanks for the details — super helpful. A few key observations from your output: - In your pool, express has priority 1 and default has priority 2. In Prefect, smaller number = higher priority, so express is higher priority than default. - default shows last_polled recently, so at least one worker is polling it. - healthy=False and late_runs_count=1 means the queue has at least one late run that is not being dispatched — typically because of a limit or eligibility issue, not because of priority. Most likely causes (and quick checks) 1) Worker subscription and capacity - Ensure your worker is polling both queues and has free capacity (e.g., it isn’t fully occupied by express runs):
Copy code
prefect worker start -p pr-worker-pool -q express -q default
- On worker startup, check logs for “Subscribed queues: …” and “Capacity …”. 2) Queue concurrency limit reached on default - default has a concurrency limit of 2. If two runs from default are already RUNNING, new ones won’t start. - Quick ways to confirm:
Copy code
prefect work-queue preview default -p pr-worker-pool --hours 1
prefect work-queue read-runs default -p pr-worker-pool
prefect flow-run ls --state RUNNING --limit 50
- As a test, you can temporarily clear the queue limit and see if runs start:
Copy code
prefect work-queue clear-concurrency-limit default -p pr-worker-pool
(You can set it back after testing.) 3) Deployment/tag/global concurrency limits - If runs in default are AwaitingConcurrencySlot due to tag/deployment/global limits, they’ll remain queued even though the queue is polled. Check run states in the UI or:
Copy code
prefect flow-run ls --limit 50
Look for states like AWAITINGCONCURRENCY or SCHEDULED vs RUNNING. 4) Queue filters - If the default queue has a filter (e.g., tags) that excludes your runs, the worker will poll but won’t get eligible work. You can review/edit queue filters in the UI for that work pool/queue. 5) Work pool state or limits - Confirm the pool itself is not paused and doesn’t have a pool-level concurrency limit that’s already saturated:
Copy code
prefect work-pool inspect pr-worker-pool
Next best steps to pinpoint the blocker Please share: - How you start your worker (the exact command) - Output of:
Copy code
prefect work-pool inspect pr-worker-pool
prefect work-queue read-runs default -p pr-worker-pool
prefect work-queue preview default -p pr-worker-pool --hours 1
prefect flow-run ls --state RUNNING --limit 50
- One flow run ID from default that is not starting With that, I can tell you precisely whether it’s queue concurrency, worker capacity, a filter/eligibility issue, or a deployment/tag/global limit causing the hold-up.