Thang Kieu
11/25/2025, 3:31 AMMarvin
11/25/2025, 3:31 AMMarvin
11/25/2025, 3:41 AMprefect work-pool set-concurrency-limit <pool-name> <limit>
- Per work queue concurrency limit:
prefect work-queue set-concurrency-limit <queue-name> <limit> --pool <pool-name>
Starting two workers on the same pool/queue (safe to do):
# Terminal 1
prefect worker start --pool my-pool --work-queue my-queue
# Terminal 2
prefect worker start --pool my-pool --work-queue my-queue
Notes and best practices:
- Multiple workers on the same queue is a common, supported way to scale horizontally.
- Prefect adds jitter to polling to avoid “thundering herd” effects.
- If a worker crashes after claiming a run (e.g., after moving it to PENDING), there’s no lease/TTL; the run may remain pending and may require intervention or resubmission.
Docs for more detail:
- Work pools
- Workers
If you can share your worker type (e.g., Docker, Kubernetes) and desired concurrency, I can suggest concrete settings.