<@ULVA73B9P> will this command help set concurrenc...
# ask-marvin
v
@Marvin will this command help set concurrency to 6 for the worker I am adding under preexisting pool?
prefect worker start --pool "remote-workers" --work-queue "queue-1" --name "worker-1" --limit 6
m
thought for 80.5 seconds
Yes. That command will start a worker named
worker-1
that polls the
remote-workers
pool (queue
queue-1
) with a concurrency limit of 6. In Prefect 3.x,
--limit
sets the maximum number of flow runs this single worker will execute concurrently. A couple notes: - The limit is per worker process at start time; it does not set a pool-wide limit. - If you start multiple workers, each worker’s
--limit
is applied independently. Docs for reference: - Workers concept - prefect worker CLI reference
v
to increase concurrency/capacity of a deployment and since a deployment can have only one worker pool, the only way is to add more workers right? each new server can be added where we run new worker under the pool that assigned to the deployment and add limit to control concurrency for worker based on server cores?
@Marvin to increase concurrency/capacity of a deployment and since a deployment can have only one worker pool, the only way is to add more workers right? each new server can be added where we run new worker under the pool that assigned to the deployment and add limit to control concurrency for worker based on server cores?
m
thought for 193.0 seconds
Short answer: yes—that’s the primary way to scale a deployment’s throughput. Key points - A deployment targets one work pool (and optionally a work queue within that pool). Throughput is determined by the total capacity of workers polling that pool/queue. - To increase capacity, either: - Start more workers for that pool/queue (on the same or additional servers), and/or - Increase each worker’s concurrency with
--limit
. - Make sure the work pool’s concurrency limit (if set) isn’t lower than your desired total capacity:
Copy code
prefect work-pool set-concurrency-limit "remote-workers" 50
- Per-deployment caps are not a separate setting. If you need a per-deployment cap, you can: - Use a dedicated work queue for that deployment and run workers only on that queue, or - Use tag-based concurrency limits on that deployment’s flow runs. Sizing tips - Process or Docker workers running flows on the same host: set
--limit
based on host resources (cores/memory). For CPU-bound workloads, start near the core count; for I/O-bound, you can go higher. - Kubernetes worker:
--limit
controls how many flow runs the worker submits/tracks concurrently; actual execution resources come from the cluster, not the worker’s host. Example: add a new worker with concurrency 6 on another server
Copy code
prefect worker start --pool "remote-workers" --work-queue "queue-1" --name "worker-2" --limit 6
Docs for deeper detail: - Workers - Work pools - Global (tag-based) concurrency limits - Work pool CLI