Fugo Takefusa
09/18/2025, 4:19 AMMarvin
09/18/2025, 4:19 AMMarvin
09/18/2025, 4:22 AMprefect.yaml
pattern for multiple deployments in Prefect 3.x. It uses YAML anchors to DRY up work pool and step definitions, supports Docker build/push, and optional Git pull/setup steps per deployment.
Notes:
- Use deployments:
(list) with unique name
and entrypoint
per deployment. You don’t need a flow_name
key.
- Use build
only if you need to build an image; push
only if you want to publish that image; pull
if your worker will clone code at runtime.
- The Docker step callables come from the prefect-docker
collection. Keep requires: prefect-docker>=0.3.1
in your steps.
- Schedules go under schedules:
(plural), and support interval
, cron
, or rrule
plus timezone
.
Example structure:
```
# Project metadata
name: prefect
prefect-version: 3.4.7
# Optional global build/push/pull; usually keep null and define per-deployment in definitions.actions
build: null
push: null
pull: null
definitions:
# Reusable work pool templates
work_pools:
docker_pool: &docker_pool
name: docker-worker-pool
work_queue_name: default
job_variables:
# Use the image produced by the build step (id: build_image)
image: '{{ build_image.image }}'
env:
PREFECT_LOGGING_LEVEL: INFO
# Reusable action step groups
actions:
# Git-based code pull (optional; use if your workers clone code at runtime)
git_pull: &git_pull
- prefect.deployments.steps.git_clone:
repository: https://github.com/your-org/your-repo
branch: main
# access_token: '{{ prefect.blocks.secret.github_token }}' # if needed
- prefect.deployments.steps.set_working_directory:
directory: your-repo
- prefect.deployments.steps.pip_install_requirements:
requirements: requirements.txt
# Docker image builds (auto-detect Dockerfile or point to a path)
docker_build_auto: &docker_build_auto
- prefect_docker.deployments.steps.build_docker_image:
id: build_image
requires: prefect-docker>=0.3.1
image_name: fugotakefusaptml/prefect-docker-repo
tag: dev
dockerfile: auto
docker_build_from_file: &docker_build_from_file
- prefect_docker.deployments.steps.build_docker_image:
id: build_image
requires: prefect-docker>=0.3.1
image_name: fugotakefusaptml/prefect-docker-repo
tag: dev2
dockerfile: data_flows/dockerfile
docker_push: &docker_push
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker>=0.3.1
image_name: '{{ build_image.image_name }}'
tag: '{{ build_image.tag }}'
deployments:
# 1) Example: build from custom Dockerfile, no push, no git pull
- name: buy-dev
entrypoint: data_flows/deploy_buy.py:buy
description: Buy flow - dev image build from Dockerfile
tags: [dev]
parameters:
dry_run: true
work_pool: *docker_pool
build: *docker_build_from_file
push: false
pull: false
schedules:
- cron: "0 6 * * *"
timezone: "UTC"
active: true
# 2) Example: auto Docker build + push to registry; scheduled twice a day
- name: orders-dev
entrypoint: data_flows/orders.py:etl
description: Orders ETL - builds and pushes image
tags: [dev]
parameters:
limit: 1000
work_pool: *docker_pool
build: *docker_build_auto
push: *docker_push
pull: false
schedules:
- cron: "0 0,12 * * *"
timezone: "UTC"
active: true
# 3) Example: no Docker build, just pull code via git at run-time on the worker
- name: reports-git
entrypoint: flows/reports/monthly.py:report_flow
description: Reports - run from git source
tags: [reports]
parameters:
month_offset: 1
work_pool:
<<: *docker_pool
# If you’re not building here, provide a static image with git + python
job_variables:Marvin
09/18/2025, 4:22 AMHow to deploy:
- From the project root, deploy all defined deployments:
prefect deploy
- Filter by name pattern:
prefect deploy --name 'orders-*'
- If you’re using Docker steps and don’t have the plugin installed globally, run with extras:
uv run --with 'prefect[docker]' prefect deploy
```
What you had vs. this template:
- Remove flow_name
under each deployment—Prefect 3.x uses entrypoint
to locate the flow function.
- Use definitions.actions
to centralize your Docker build/push and Git pull steps.
- Keep job_variables.image: '{{ build_image.image }}'
so Docker workers run the freshly built image.
- Use schedules:
(plural) for one or more schedule entries; each supports cron
, interval
, or rrule
and timezone
.
Docs and references:
- Prefect YAML guide: prefect.yaml
- Schedules in Prefect 3: Schedules
- Storing/pulling flow code: Store flow code
- Docker step functions: prefect-docker steps
If you share your target registry/image naming and whether your workers will clone code or run from image-only, I can tailor the template exactly to your setup.