alex joyce
09/19/2025, 8:59 AMMarvin
09/19/2025, 8:59 AMMarvin
09/19/2025, 9:06 AMprefect worker start --pool "my-pool" --limit 5
- Python (equivalent):
from prefect.workers.process import ProcessWorker
ProcessWorker(work_pool_name="my-pool", limit=5).start()
- If you don’t set a limit, the worker won’t enforce a cap itself; your machine resources and any pool/global limits will be the bottleneck.
- Docs: Workers concept
- Work pool concurrency
- Caps the total number of running flow runs across all workers in the pool.
- If not set, there’s no pool-level cap.
- Set or clear:
prefect work-pool set-concurrency-limit "my-pool" 10
prefect work-pool clear-concurrency-limit "my-pool"
- Docs: Work pools concept and work-pool CLI
- Task-level (tag-based) concurrency limits
- Optional throttle on tasks (or flows if you apply the tag to flows) by tag. Only needed if you want to restrict specific workloads.
- Create:
prefect concurrency-limit create my-tag 3
- Docs: Global/Tag concurrency limits
Putting it together
- To run flows simultaneously:
- Start at least one worker for your work pool. If you want parallelism on a single worker, give it a higher --limit
(e.g., 5).
- Make sure your work pool concurrency limit is not set too low (or unset if you don’t need a cap).
- Only use tag-based concurrency limits if you want to intentionally throttle certain runs.
Questions to tailor the advice:
- Which worker type are you using (e.g., Process, Docker, Kubernetes)?
- Are runs currently queuing, or are you just planning capacity?
- Do you want a strict cap on parallelism, or do you just want “as many as resources allow”?
Helpful references:
- Workers: Workers
- Work pools and their concurrency limit: Work pools
- Global/Tag concurrency: Global concurrency limits