Victor
08/28/2025, 4:16 PMprefect worker start --pool "remote-workers" --work-queue "queue-1" --name "worker-1" --limit 6
Marvin
08/28/2025, 4:16 PMMarvin
08/28/2025, 4:17 PMworker-1
that polls the remote-workers
pool (queue queue-1
) with a concurrency limit of 6. In Prefect 3.x, --limit
sets the maximum number of flow runs this single worker will execute concurrently.
A couple notes:
- The limit is per worker process at start time; it does not set a pool-wide limit.
- If you start multiple workers, each worker’s --limit
is applied independently.
Docs for reference:
- Workers concept
- prefect worker CLI referenceVictor
08/28/2025, 4:24 PMVictor
08/28/2025, 4:27 PMMarvin
08/28/2025, 4:27 PMMarvin
08/28/2025, 4:31 PM--limit
.
- Make sure the work pool’s concurrency limit (if set) isn’t lower than your desired total capacity:
prefect work-pool set-concurrency-limit "remote-workers" 50
- Per-deployment caps are not a separate setting. If you need a per-deployment cap, you can:
- Use a dedicated work queue for that deployment and run workers only on that queue, or
- Use tag-based concurrency limits on that deployment’s flow runs.
Sizing tips
- Process or Docker workers running flows on the same host: set --limit
based on host resources (cores/memory). For CPU-bound workloads, start near the core count; for I/O-bound, you can go higher.
- Kubernetes worker: --limit
controls how many flow runs the worker submits/tracks concurrently; actual execution resources come from the cluster, not the worker’s host.
Example: add a new worker with concurrency 6 on another server
prefect worker start --pool "remote-workers" --work-queue "queue-1" --name "worker-2" --limit 6
Docs for deeper detail:
- Workers
- Work pools
- Global (tag-based) concurrency limits
- Work pool CLI