<@ULVA73B9P> - what is the typical deployment time...
# ask-marvin
j
@Marvin - what is the typical deployment time for a single deployment? what about 100 deployments?
m
thought for 179.1 seconds
Short answer: there isn’t a fixed “typical” time because it depends heavily on what your deploy step is doing. Here are practical ranges and what drives them (assuming Prefect 3.x): What we commonly see - Single deployment, registration-only (no Docker build/push, reusing an existing image or process worker): usually a few seconds per deployment (often 2–10s) to import the flow and register it with the API. - Single deployment with Docker image build/push: dominated by the build/push. With warm cache: ~15–60s; cold/full build: ~1–5+ minutes depending on image size and registry speed. - 100 deployments run sequentially: - Registration-only: roughly linear, often in the 3–15 minute range in total. - If each one builds/pushes its own image: multiply the per-deployment build time by 100 — this can easily become hours. Best practice is to build once and reuse. What affects deployment time - Docker image builds/pushes (layer cache, base image size, registry throughput) - How heavy your flow import is (imports that do work at import-time slow deploys) - Network latency and API/database load (Prefect Cloud vs self-hosted, region) - Any source-code checks your process performs during deploy (e.g., pinning a commit) Tips to speed it up - Pre-build a single image in CI and reference it across deployments; don’t build in the deploy step unless you have to. - Keep flow imports lightweight (avoid heavy work at import time). - Group definitions and run one deploy command; it still registers sequentially but reduces command overhead. - If you must deploy many at once, batch them and avoid overwhelming your Prefect API/database; parallelization helps only if you’re not building/pushing and your API can handle it. Real-world datapoint - An enterprise case study reduced “hundreds of workflows” deployment from hours to ~20–30 minutes by eliminating per-deployment builds and tuning concurrency/infrastructure: Case study Docs for reference - prefect.yaml and multi-deployment projects - Deploy via Python (flow.deploy) - Storing/serving your flow code If you share your setup (Cloud or self-hosted, whether you build Docker images during deploy, and roughly how big your image/repo is), I can give a tighter estimate for 1 vs 100 deployments.
j
I'm only referencing deploying the deployments using
prefect deploy
we will use one docker image for all deployments. If we had hundreds of deployments though - how long would it take
@Marvin I'm only referencing deploying the deployments using
prefect deploy
we will use one docker image for all deployments. If we had hundreds of deployments though - how long would it take
m
thought for 66.5 seconds
Great—since you’re using one pre-built Docker image for all deployments,
prefect deploy
won’t spend time building/pushing images. In that case, deploy time is mostly: - importing each flow - validating config - registering the deployment with the API Rule of thumb (Prefect 3.x) - Per deployment: typically 1–5 seconds if imports are light and the network/API is healthy - 100 deployments: roughly 2–10 minutes, linearly scaling with count - If imports are heavy (big libraries, top-level work at import): 5–15 seconds each → 8–25 minutes for 100 For “hundreds,” you can estimate linearly. Example: - 300 deployments at ~3s each → ~15 minutes - 300 deployments at ~8s each → ~40 minutes How to get a quick, accurate estimate in your environment - Time a subset and extrapolate. For example, deploy 10 by pattern:
Copy code
prefect deploy --name 'my-flow-prefix/*'
If that takes ~30s, you’re ~3s/deployment → 100 ≈ 5 minutes. You can deploy all at once with:
Copy code
prefect deploy --all
Tips to keep it fast - Keep flow imports lightweight (avoid heavy work at module import time). - Consolidate deployments in one project and deploy in a single command to minimize overhead. - If you do parallelize multiple deploy commands, be mindful of API load; it’s usually best to keep to a small concurrency. References - prefect.yaml and multi-deployment projects - Deploy via Python (flow.deploy) - Case study on cutting hundreds-of-deployments time by eliminating per-deployment builds: enterprise case study If you can share roughly how heavy your imports are (or a timing from 10 deployments), I can give a tighter estimate for “hundreds.”
j
@Marvin - is there a way to selectively deploy certain deployments based on a filter? For example, if only a single flow changed - I would only want to deploy the deployments under that flow
m
thought for 42.9 seconds
Yes—using prefect deploy you can filter which deployments to register. Two handy filters: - By name pattern (matches flow_name/deployment_name) - All deployments for a single flow:
Copy code
prefect deploy --name 'my_flow/*'
- A single deployment:
Copy code
prefect deploy --name 'my_flow/prod'
- Wildcards work (e.g.,
*-flow/*
,
*/staging-*
) - By entrypoint (file:function) to target just that flow’s deployments:
Copy code
prefect deploy ./flows/my_flow.py:my_flow
You can combine these if you want to be extra specific (e.g., entrypoint + a name pattern). CI tip for “only deploy what changed” - Detect changed flows and deploy only those:
Copy code
for f in $(git diff --name-only origin/main...HEAD | grep '^flows/.*\.py$'); do
  flow_func=$(basename "$f" .py)
  prefect deploy "$f:$flow_func"
done
Or, if your deployments are named
flow/deployment
, map changed flow to a name pattern:
Copy code
prefect deploy --name 'my_flow/*'
Docs - prefect.yaml and multi-deployment projects - CLI help shows examples for the --name pattern; run:
Copy code
prefect deploy --help
j
@Marvin - does that cli command work if you have multiple different
prefect.yaml
files? Can you specify each
prefect.yaml
?
m
thought for 281.9 seconds
Yes. You can target a specific project file with
--prefect-file
, so multiple
prefect.yaml
files are fine—just run one command per file. Examples - Deploy all deployments from a specific file:
Copy code
prefect deploy --prefect-file services/payments/prefect.yaml --all
- Deploy only certain deployments from that file (by name pattern):
Copy code
prefect deploy --prefect-file services/payments/prefect.yaml --name 'payments-flow/*'
- Deploy a single flow by entrypoint (within that file’s project):
Copy code
prefect deploy --prefect-file services/payments/prefect.yaml ./flows/payments.py:payments_flow
Notes - You cannot pass multiple
--prefect-file
values in a single command. Run one
prefect deploy
per file. - Filtering with
--name
applies to the deployments defined in the file you point to. -
--all
deploys all deployments defined in that specific file. - Run the command from your project root so entrypoints resolve correctly (or use absolute paths). CI tip (deploy only changed project files)
Copy code
for f in $(git diff --name-only origin/main...HEAD | grep 'prefect\.yaml$'); do
  prefect deploy --prefect-file "$f" --all --no-prompt
done
Docs - prefect.yaml and multi-deployment projects