Great question! Short version: a work pool is the top-level execution boundary and infrastructure configuration; a work queue is a routing/priority/concurrency partition inside a single work pool.
Here’s how to think about them in Prefect 3.x:
- Work pools
- What they are: The execution backend “container” for your deployments. Each pool is typed (e.g., Kubernetes, Docker, Process) and defines the base job template and infra settings.
- What they’re for: Separating different infrastructures, environments, or security boundaries. Pools have their own concurrency limits and configuration.
- You run workers against a pool. Workers provision/run your flow-run jobs using that pool’s base job template.
- Docs:
Work pools concept
- Work queues
- What they are: Named sub-partitions within a pool used for routing and scheduling controls.
- What they’re for: Prioritization and throttling within the same infrastructure. Queues support:
- Concurrency limits per queue
- Priority ordering across queues in the same pool
- Deployments can target a specific work queue within their pool; workers can be configured to poll all queues in a pool or specific queue(s).
- CLI:
work-queue CLI
When to use multiple work pools vs multiple work queues
- Use multiple work pools when:
- You need different infrastructure backends or base job templates (e.g., Kubernetes vs Docker; different k8s clusters/node pools)
- You want environment isolation (dev/stage/prod), tighter security boundaries, or separate resource governance
- You need different default job settings, images, or credentials at the pool level
- Use multiple work queues (inside one pool) when:
- You want to prioritize some runs over others (e.g., high vs low priority)
- You want separate concurrency controls without changing infrastructure
- You want simple logical routing within the same infra (e.g., “backfill” vs “realtime” queues)
How it fits together at runtime
- A deployment specifies
work_pool_name
and optionally
work_queue_name
- A worker polls a pool (and optionally specific queues) and launches jobs with the pool’s configuration
- Concurrency can be set at both pool and queue levels; priority applies at the queue level within a pool
Useful CLI (verified)
- List and create pools
prefect work-pool ls
prefect work-pool create "k8s-prod" --type kubernetes --paused
- Start a worker for a pool (optionally specific queues)
prefect worker start --pool k8s-prod
prefect worker start --pool k8s-prod --work-queue high --work-queue default
- Manage queues within a pool
prefect work-queue ls
prefect work-queue create high --pool k8s-prod --priority 1
prefect work-queue set-concurrency-limit high --limit 10
Notes on Prefect 2.x vs 3.x
- 2.x used agents + work queues. 3.x uses workers + work pools (workers replace agents).
- Work queues still exist in 3.x but are always scoped to a pool.
- If you’re migrating: your 2.x “agent+queue” typically maps to a 3.x “worker+pool(+queue)”.
References
- Work pools:
Concepts: Work pools
- Work queue CLI:
CLI: work-queue
- Work pool CLI:
CLI: work-pool
- Blog overview:
Work pools & workers blog
If you can share your setup (infra types, environments, and what you want to prioritize/throttle), I can suggest a concrete pool/queue layout.