Samo K.
09/28/2025, 10:38 PMMarvin
09/28/2025, 10:38 PMMarvin
09/28/2025, 10:47 PMprefect worker start -p my-docker-pool -q host-a --limit 1
- Start one such worker on each host (e.g., host-a, host-b). Setting --limit 1
ensures only one container/run at a time on that host. Increase the number to allow more.
- Docs: Work pools and workers
- Pool- or queue-wide caps (global across hosts):
- Pool-wide cap (across all hosts/workers in the pool):
prefect work-pool set-concurrency-limit "my-docker-pool" 5
- Queue-specific cap (useful if you create one queue per host and have the host’s worker only poll that queue):
prefect work-queue create host-a --pool my-docker-pool --limit 1
# or, to set after creation
prefect work-queue set-concurrency-limit host-a 1 --pool my-docker-pool
- CLI help references:
- work-pool CLI
- work-queue CLI
- One-at-a-time per flow (any host): if you mean “this flow should never run more than once concurrently anywhere”, use a global concurrency limit via a tag.
prefect concurrency-limit create my-flow-serial 1
Then add the my-flow-serial
tag to your deployment so all runs of that flow share the same global slot. Docs: Global concurrency limits
- Pinning a flow to a specific host: run a dedicated worker on that host polling a dedicated work queue (e.g., host-a
) and set the deployment to use that queue. That routes all runs of that deployment to that host. You can combine this with --limit 1
on the host’s worker.
Notes:
- A single flow run always executes on one host/container by default (tasks are not spread across hosts unless you explicitly use a distributed executor inside your flow).
- If you accidentally run multiple worker processes on the same host, each with --limit 1
, you could end up with more than one run per host. Either enforce a single worker per host or combine with a per-host queue capped to 1.
Quick sanity examples:
- One-at-a-time per host:
- Host A: prefect worker start -p my-docker-pool -q host-a --limit 1
- Host B: prefect worker start -p my-docker-pool -q host-b --limit 1
- One-at-a-time globally for a flow:
- prefect concurrency-limit create my-flow-serial 1
- Tag the deployment with my-flow-serial
.
Would you like help setting up per-host queues and updating your deployment(s) to target them, or do you want a global “only one run anywhere” constraint for a specific flow?