<@ULVA73B9P> what is the difference between having...
# ask-marvin
a
@Marvin what is the difference between having multiple work pools and multiple work queues
m
thought for 189.1 seconds
Great question! Short version: a work pool is the top-level execution boundary and infrastructure configuration; a work queue is a routing/priority/concurrency partition inside a single work pool. Here’s how to think about them in Prefect 3.x: - Work pools - What they are: The execution backend “container” for your deployments. Each pool is typed (e.g., Kubernetes, Docker, Process) and defines the base job template and infra settings. - What they’re for: Separating different infrastructures, environments, or security boundaries. Pools have their own concurrency limits and configuration. - You run workers against a pool. Workers provision/run your flow-run jobs using that pool’s base job template. - Docs: Work pools concept - Work queues - What they are: Named sub-partitions within a pool used for routing and scheduling controls. - What they’re for: Prioritization and throttling within the same infrastructure. Queues support: - Concurrency limits per queue - Priority ordering across queues in the same pool - Deployments can target a specific work queue within their pool; workers can be configured to poll all queues in a pool or specific queue(s). - CLI: work-queue CLI When to use multiple work pools vs multiple work queues - Use multiple work pools when: - You need different infrastructure backends or base job templates (e.g., Kubernetes vs Docker; different k8s clusters/node pools) - You want environment isolation (dev/stage/prod), tighter security boundaries, or separate resource governance - You need different default job settings, images, or credentials at the pool level - Use multiple work queues (inside one pool) when: - You want to prioritize some runs over others (e.g., high vs low priority) - You want separate concurrency controls without changing infrastructure - You want simple logical routing within the same infra (e.g., “backfill” vs “realtime” queues) How it fits together at runtime - A deployment specifies
work_pool_name
and optionally
work_queue_name
- A worker polls a pool (and optionally specific queues) and launches jobs with the pool’s configuration - Concurrency can be set at both pool and queue levels; priority applies at the queue level within a pool Useful CLI (verified) - List and create pools
Copy code
prefect work-pool ls
prefect work-pool create "k8s-prod" --type kubernetes --paused
- Start a worker for a pool (optionally specific queues)
Copy code
prefect worker start --pool k8s-prod
prefect worker start --pool k8s-prod --work-queue high --work-queue default
- Manage queues within a pool
Copy code
prefect work-queue ls
prefect work-queue create high --pool k8s-prod --priority 1
prefect work-queue set-concurrency-limit high --limit 10
Notes on Prefect 2.x vs 3.x - 2.x used agents + work queues. 3.x uses workers + work pools (workers replace agents). - Work queues still exist in 3.x but are always scoped to a pool. - If you’re migrating: your 2.x “agent+queue” typically maps to a 3.x “worker+pool(+queue)”. References - Work pools: Concepts: Work pools - Work queue CLI: CLI: work-queue - Work pool CLI: CLI: work-pool - Blog overview: Work pools &amp; workers blog If you can share your setup (infra types, environments, and what you want to prioritize/throttle), I can suggest a concrete pool/queue layout.