Tom Han
09/09/2025, 10:25 PMMarvin
09/09/2025, 10:25 PMMarvin
09/09/2025, 10:34 PMprefect work-pool set-concurrency-limit <your-pool> 1
This caps concurrency across all workers in that pool.
- Limit a specific work queue (useful if you run one queue per worker):
prefect work-queue set-concurrency-limit <your-queue> -p <your-pool> 1
This caps concurrency for that queue only.
Notes and gotchas:
- These limits apply immediately for new starts; any flow runs already in progress will continue.
- A pool limit of 1 will restrict the entire pool to a single concurrent flow run, not “one per worker.” If you need one-per-worker without restarting them, the usual pattern is one queue per worker and set each queue’s limit to 1.
- If you can restart workers, you can set a per-worker limit when starting them (so each worker only starts one flow at a time). Example:
prefect worker start -p <your-pool> --name <your-worker-name> --limit 1
- For broader controls, you can also use tag-based concurrency limits to cap concurrent runs across specific flows/tasks regardless of pool/queue:
prefect concurrency-limit create <tag> <limit>
Docs:
- Workers
- Work pools
- Concurrency limits
If you share how your workers are organized (how many, which pools/queues), I can suggest the least disruptive way to enforce “one flow at a time per worker” right now.