Short answer: because pools define “how to run,” queues define “what to run next.” Mixing them would make life… needlessly repetitive.
- Work pool = infrastructure boundary + worker type
- Holds the worker type (e.g., Kubernetes, Docker, ECS, Process)
- Centralizes the base job template: image, cluster/namespace, resources, env, credentials, etc.
- Lets you change infra defaults in one place and have every deployment picking that pool inherit it (with per-deployment overrides when needed)
- Multiple workers can attach to the same pool across machines/clusters
- Work queue (inside a pool) = routing and rate control
- Prioritization and concurrency limits
- Separate lanes like prod/staging/batch without duplicating infra config
- Deployments target a pool and may specify a queue; workers pull from queues in their pool
Why not “just a queue per infra”? You’d:
- Duplicate the same infra config across many queues
- Lose the ability to swap infra centrally without redeploying
- Conflate routing/backpressure concerns with infrastructure
- Make it harder to share infra across teams while keeping separate priorities/limits
Concrete example:
- One Kubernetes pool (namespace=prod, base image, service account)
- Queues: high-priority (concurrency 5), default (concurrency 20), batch (concurrency 50)
- If you need to bump the base image or change the namespace, do it once at the pool; all queues and deployments inherit it.
If you’re coming from Prefect 2.x: workers replace agents, and pools are first-class; deployments target pools (and optionally queues). The “queue-only” mental model was always a bit cramped for real-world infra governance.
Docs and references:
-
Work pools (concepts)
-
Work queue CLI reference
-
Blog: Work pools and workers — deploy Python securely
If you share your infra target (e.g., K8s vs ECS) and how you want to prioritize traffic, I can suggest a clean pool/queue layout. I promise it’ll be less depressing than it sounds.