Alastair
12/19/2025, 4:38 AMMarvin
12/19/2025, 4:38 AMMarvin
12/19/2025, 4:45 AM# prefect.yaml
definitions:
shared: &shared
entrypoint: flows/my_flow.py:my_flow
parameters:
log_level: INFO
local_pool: &local_pool
name: local
docker_pool: &docker_pool
name: docker-prod
job_variables:
image: "{{ build-image.image }}"
local_pull: &local_pull
- prefect.deployments.steps.set_working_directory:
directory: .
git_pull: &git_pull
- prefect.deployments.steps.git_clone:
repository: "<https://github.com/your-org/your-repo.git>"
branch: main
# Global build step (used by the docker deployment)
build:
- prefect_docker.deployments.steps.build_docker_image:
id: build-image
requires: prefect-docker>=0.5.0
image_name: your-repo/your-image
tag: "{{ $IMAGE_TAG | default('dev') }}"
dockerfile: auto
deployments:
# Local dev: run with local worker, use working dir as source
- name: my-flow-dev
<<: *shared
tags: [dev]
work_pool: *local_pool
pull: *local_pull
build: null # skip docker build for dev
push: null # skip docker push
# Cloud/prod: docker worker, pull code from git, push image
- name: my-flow-prod
<<: *shared
tags: [prod]
work_pool: *docker_pool
pull: *git_pull
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker>=0.5.0
image_name: "{{ build-image.image_name }}"
tag: "{{ build-image.tag }}"
Deploy either one without duplicating the whole file:
prefect deploy -n my-flow-dev
prefect deploy -n my-flow-prod
Notes and options:
- This keeps one source of truth and only varies the pull and work_pool sections.
- If you want to avoid hardcoding environment-specific values, parameterize with env vars in templating (e.g., {{ $IMAGE_TAG }}, {{ $GIT_BRANCH }}) and set them in CI/CD.
- If you truly need two separate files, you can do that too and select with --prefect-file, but the single-file + anchors approach usually scales better.
- Make sure you have the docker integration installed for the docker steps (e.g., pip install prefect-docker).
Helpful docs:
- Prefect YAML and templating: prefect.yaml guide
- Create deployments: Create deployments
- Pull steps (git_clone, set_working_directory): Pull steps API
- Docker deployments: Docker infrastructure guide
- Work pools and workers: Work pools concept and Workers concept
- CLI reference for prefect deploy: prefect deploy CLI
If you’d like, share your current prefect.yaml and I can help refactor it to this pattern.Alastair
12/19/2025, 4:47 AMpullMarvin
12/19/2025, 4:47 AMMarvin
12/19/2025, 4:53 AM# steps/conditional_pull.py
import os
from typing import Optional, Dict
from prefect.deployments.steps.pull import git_clone
async def conditional_pull(
repository: Optional[str] = None,
branch: Optional[str] = None,
commit_sha: Optional[str] = None,
directory: str = ".",
env_var: str = "PREFECT_LOCAL_DEV",
) -> Dict[str, str]:
# If set for local dev, just use the working directory
if os.getenv(env_var, "").lower() in {"1", "true", "yes"}:
return {"directory": directory}
# Otherwise clone from the repo
if repository:
return await git_clone(repository=repository, branch=branch, commit_sha=commit_sha)
# Fallback to current directory
return {"directory": directory}
prefect.yaml:
pull:
- steps.conditional_pull:
repository: "<https://github.com/your-org/your-repo.git>"
branch: "main"
directory: "."
env_var: "PREFECT_LOCAL_DEV"
- Run local workers with PREFECT_LOCAL_DEV=1 so they use your local code; prod workers omit it and clone from git.
- Docs for pull steps: Pull steps API
- YAML guide: prefect.yaml guide
2) Two files, zero duplication of deployments via overlays (only pull differs)
- Keep a base file with all deployments. Keep tiny overlays for dev/prod that only set the top-level pull.
- In CI, merge the base + overlay into a working prefect.yaml before prefect deploy.
Example:
# prefect.base.yaml (only deployments; no pull)
deployments:
- name: flow-a
entrypoint: flows/a.py:a
work_pool: { name: my-pool }
- name: flow-b
entrypoint: flows/b.py:b
work_pool: { name: my-pool }
# ... 10 flows here
# prefect.overlay.dev.yaml
pull:
- prefect.deployments.steps.set_working_directory:
directory: .
# prefect.overlay.prod.yaml
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/your-org/your-repo.git>
branch: main
Merge and deploy:
yq ea '. as $item ireduce ({}; . * $item )' prefect.base.yaml prefect.overlay.dev.yaml > prefect.yaml
prefect deploy --all
- This keeps the deployments list in one place, and only the pull lives in the overlay files.
3) Generate dev/prod deployments with a small Python script
- If you’re okay using the Python API, keep a single list of flows and programmatically create both deployments per flow.
- This centralizes the “duplication” as a loop, not repeated YAML blocks.
- Flow API: Flow API
Notes:
- There’s no built-in CLI or env var to “disable pull” on an existing deployment; you either define different pull steps, or make them conditional as in option 1.
- If dev and prod require different work pools (e.g., local vs docker), you do need two deployments per flow. Options 2 or 3 will still keep maintenance light.
If you can confirm whether your work pools differ between dev and prod, I can tailor one of these patterns with exact snippets for your repo structure.Alastair
12/19/2025, 4:54 AMMarvin
12/19/2025, 4:54 AMMarvin
12/19/2025, 4:54 AM