<@ULVA73B9P> why there is a concept of workpool wh...
# ask-marvin
r
@Marvin why there is a concept of workpool when it can be just queue dedicatd to a specific infra?
m
thought for 124.7 seconds
Short answer: because pools define “how to run,” queues define “what to run next.” Mixing them would make life… needlessly repetitive. - Work pool = infrastructure boundary + worker type - Holds the worker type (e.g., Kubernetes, Docker, ECS, Process) - Centralizes the base job template: image, cluster/namespace, resources, env, credentials, etc. - Lets you change infra defaults in one place and have every deployment picking that pool inherit it (with per-deployment overrides when needed) - Multiple workers can attach to the same pool across machines/clusters - Work queue (inside a pool) = routing and rate control - Prioritization and concurrency limits - Separate lanes like prod/staging/batch without duplicating infra config - Deployments target a pool and may specify a queue; workers pull from queues in their pool Why not “just a queue per infra”? You’d: - Duplicate the same infra config across many queues - Lose the ability to swap infra centrally without redeploying - Conflate routing/backpressure concerns with infrastructure - Make it harder to share infra across teams while keeping separate priorities/limits Concrete example: - One Kubernetes pool (namespace=prod, base image, service account) - Queues: high-priority (concurrency 5), default (concurrency 20), batch (concurrency 50) - If you need to bump the base image or change the namespace, do it once at the pool; all queues and deployments inherit it. If you’re coming from Prefect 2.x: workers replace agents, and pools are first-class; deployments target pools (and optionally queues). The “queue-only” mental model was always a bit cramped for real-world infra governance. Docs and references: - Work pools (concepts) - Work queue CLI reference - Blog: Work pools and workers — deploy Python securely If you share your infra target (e.g., K8s vs ECS) and how you want to prioritize traffic, I can suggest a clean pool/queue layout. I promise it’ll be less depressing than it sounds.
n
work pools define the infra config for work (e.g. k8s pods per flow run with X mem and Y CPU) work queues are priority lanes with optional concurrency limits within a work pool i.e. you might want all your training jobs to run on k8s but certain deployments can only run 3 at a time and certain deployments are more important to run first
marvin's answer is pretty solid
r
got it. thank you