<@ULVA73B9P> I am using Prefect with a Docker work...
# ask-marvin
s
@Marvin I am using Prefect with a Docker workpool. Can I control how many Docker containers/workers may run within a single host? I want to make sure each flow only runs on 1 physical host at a time
m
thought for 547.3 seconds
Short answer: yes. In Prefect 3.x with a Docker work pool, you can cap how many flow runs (containers) start per host and you can also ensure runs are serialized either per host or globally. Here are the common patterns: - Per-host cap (recommended): run one Docker worker process per host and give it a local limit. The worker only starts containers on the host it’s running on, so this effectively caps containers per host.
Copy code
prefect worker start -p my-docker-pool -q host-a --limit 1
- Start one such worker on each host (e.g., host-a, host-b). Setting
--limit 1
ensures only one container/run at a time on that host. Increase the number to allow more. - Docs: Work pools and workers - Pool- or queue-wide caps (global across hosts): - Pool-wide cap (across all hosts/workers in the pool):
Copy code
prefect work-pool set-concurrency-limit "my-docker-pool" 5
- Queue-specific cap (useful if you create one queue per host and have the host’s worker only poll that queue):
Copy code
prefect work-queue create host-a --pool my-docker-pool --limit 1
    # or, to set after creation
    prefect work-queue set-concurrency-limit host-a 1 --pool my-docker-pool
- CLI help references: - work-pool CLI - work-queue CLI - One-at-a-time per flow (any host): if you mean “this flow should never run more than once concurrently anywhere”, use a global concurrency limit via a tag.
Copy code
prefect concurrency-limit create my-flow-serial 1
Then add the
my-flow-serial
tag to your deployment so all runs of that flow share the same global slot. Docs: Global concurrency limits - Pinning a flow to a specific host: run a dedicated worker on that host polling a dedicated work queue (e.g.,
host-a
) and set the deployment to use that queue. That routes all runs of that deployment to that host. You can combine this with
--limit 1
on the host’s worker. Notes: - A single flow run always executes on one host/container by default (tasks are not spread across hosts unless you explicitly use a distributed executor inside your flow). - If you accidentally run multiple worker processes on the same host, each with
--limit 1
, you could end up with more than one run per host. Either enforce a single worker per host or combine with a per-host queue capped to 1. Quick sanity examples: - One-at-a-time per host: - Host A:
prefect worker start -p my-docker-pool -q host-a --limit 1
- Host B:
prefect worker start -p my-docker-pool -q host-b --limit 1
- One-at-a-time globally for a flow: -
prefect concurrency-limit create my-flow-serial 1
- Tag the deployment with
my-flow-serial
. Would you like help setting up per-host queues and updating your deployment(s) to target them, or do you want a global “only one run anywhere” constraint for a specific flow?