Ishan Anilbhai Koradiya
09/23/2025, 11:49 AMMarvin
09/23/2025, 11:49 AMMarvin
09/23/2025, 11:56 AMprefect work-pool create my-process-pool --type process
2) Start workers on separate machines (point them at the same pool and optional queues)
# Machine A
prefect worker start -p my-process-pool -q default
# Machine B
prefect worker start -p my-process-pool -q default
Each worker will poll for work and “pull” flow runs—this provides natural load distribution without a load balancer.
3) Control concurrency
- Per-pool limit (caps total concurrent runs across all workers in the pool):
prefect work-pool set-concurrency-limit my-process-pool 10
- Per-queue limit (caps concurrent runs for a specific queue in that pool):
prefect work-queue set-concurrency-limit <your-queue-name> 5
- Per-worker limit: you can cap how many runs a single worker will execute concurrently (useful when you know a machine’s capacity).
Notes on load balancers:
- Not needed for process workers—they initiate outbound polling to Prefect and don’t expose an HTTP service.
- If you are self-hosting the Prefect API/Server, you can absolutely put the API behind a load balancer, but that’s separate from the workers themselves.
Docs and references:
- Workers concepts
- Process worker API
- Global concurrency limits
- Background: Work pools and workers blog
If you share a bit about your environment (Prefect Cloud vs. self-hosted, OS, and expected concurrency), I can suggest concrete settings and a rollout plan for your workers.