<@ULVA73B9P> suggest me the nice prefect yml struc...
# ask-marvin
f
@Marvin suggest me the nice prefect yml structure when you have multiple deployments: # Welcome to your prefect.yaml file! You can use this file for storing and managing # configuration for deploying your flows. We recommend committing this file to source # control along with your flow code. # Generic metadata about this project name: prefect prefect-version: 3.4.7 # build section allows you to manage and build docker images build: null # - prefect_docker.deployments.steps.build_docker_image: # id: build_image # requires: prefect-docker>=0.3.1 # image_name: fugotakefusaptml/prefect-docker-repo # tag: 2 # dockerfile: auto # push section allows you to manage if and how this project is uploaded to remote locations push: null # - prefect_docker.deployments.steps.push_docker_image: # requires: prefect-docker>=0.3.1 # image_name: '{{ build_image.image_name }}' # tag: '{{ build_image.tag }}' # pull section allows you to provide instructions for cloning this project in remote locations pull: null # - prefect.deployments.steps.set_working_directory: # directory: /opt/prefect/prefect definitions: work_pools: docker_worker_pool: &docker_worker_pool name: docker-worker-pool work_queue_name: null job_variables: image: '{{ build_image.image }}' actions: docker_build: &docker_build - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.3.0 image_name: fugotakefusaptml/prefect-docker-repo tag: dev dockerfile: auto docker_build_docker_file: &docker_build_docker_file - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.3.0 image_name: fugotakefusaptml/prefect-docker-repo tag: dev2 dockerfile: 'data_flows/dockerfile' docker_push: &docker_push - prefect_docker.deployments.steps.push_docker_image: requires: prefect-docker>=0.3.1 image_name: fugotakefusaptml/prefect-docker-repo tag: dev # the deployments section allows you to provide configuration for deploying flows deployments: - name: deploy_buy1 flow_name: deploybuy entrypoint: data_flows/deploy_buy.py:buy work_pool: *docker_worker_pool build: *docker_build_docker_file push: false pull: false # push: *docker_push
m
thought for 218.9 seconds
Great question. Here’s a clean, reusable
prefect.yaml
pattern for multiple deployments in Prefect 3.x. It uses YAML anchors to DRY up work pool and step definitions, supports Docker build/push, and optional Git pull/setup steps per deployment. Notes: - Use
deployments:
(list) with unique
name
and
entrypoint
per deployment. You don’t need a
flow_name
key. - Use
build
only if you need to build an image;
push
only if you want to publish that image;
pull
if your worker will clone code at runtime. - The Docker step callables come from the
prefect-docker
collection. Keep
requires: prefect-docker>=0.3.1
in your steps. - Schedules go under
schedules:
(plural), and support
interval
,
cron
, or
rrule
plus
timezone
. Example structure: ``` # Project metadata name: prefect prefect-version: 3.4.7 # Optional global build/push/pull; usually keep null and define per-deployment in definitions.actions build: null push: null pull: null definitions: # Reusable work pool templates work_pools: docker_pool: &docker_pool name: docker-worker-pool work_queue_name: default job_variables: # Use the image produced by the build step (id: build_image) image: '{{ build_image.image }}' env: PREFECT_LOGGING_LEVEL: INFO # Reusable action step groups actions: # Git-based code pull (optional; use if your workers clone code at runtime) git_pull: &git_pull - prefect.deployments.steps.git_clone: repository: https://github.com/your-org/your-repo branch: main # access_token: '{{ prefect.blocks.secret.github_token }}' # if needed - prefect.deployments.steps.set_working_directory: directory: your-repo - prefect.deployments.steps.pip_install_requirements: requirements: requirements.txt # Docker image builds (auto-detect Dockerfile or point to a path) docker_build_auto: &docker_build_auto - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.3.1 image_name: fugotakefusaptml/prefect-docker-repo tag: dev dockerfile: auto docker_build_from_file: &docker_build_from_file - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.3.1 image_name: fugotakefusaptml/prefect-docker-repo tag: dev2 dockerfile: data_flows/dockerfile docker_push: &docker_push - prefect_docker.deployments.steps.push_docker_image: requires: prefect-docker>=0.3.1 image_name: '{{ build_image.image_name }}' tag: '{{ build_image.tag }}' deployments: # 1) Example: build from custom Dockerfile, no push, no git pull - name: buy-dev entrypoint: data_flows/deploy_buy.py:buy description: Buy flow - dev image build from Dockerfile tags: [dev] parameters: dry_run: true work_pool: *docker_pool build: *docker_build_from_file push: false pull: false schedules: - cron: "0 6 * * *" timezone: "UTC" active: true # 2) Example: auto Docker build + push to registry; scheduled twice a day - name: orders-dev entrypoint: data_flows/orders.py:etl description: Orders ETL - builds and pushes image tags: [dev] parameters: limit: 1000 work_pool: *docker_pool build: *docker_build_auto push: *docker_push pull: false schedules: - cron: "0 0,12 * * *" timezone: "UTC" active: true # 3) Example: no Docker build, just pull code via git at run-time on the worker - name: reports-git entrypoint: flows/reports/monthly.py:report_flow description: Reports - run from git source tags: [reports] parameters: month_offset: 1 work_pool: <<: *docker_pool # If you’re not building here, provide a static image with git + python job_variables:
image: your-registry/py-runner:3.11 build: false push: false pull: *git_pull
Copy code
How to deploy:
- From the project root, deploy all defined deployments:
prefect deploy
Copy code
- Filter by name pattern:
prefect deploy --name 'orders-*'
Copy code
- If you’re using Docker steps and don’t have the plugin installed globally, run with extras:
uv run --with 'prefect[docker]' prefect deploy ``` What you had vs. this template: - Remove
flow_name
under each deployment—Prefect 3.x uses
entrypoint
to locate the flow function. - Use
definitions.actions
to centralize your Docker build/push and Git pull steps. - Keep
job_variables.image: '{{ build_image.image }}'
so Docker workers run the freshly built image. - Use
schedules:
(plural) for one or more schedule entries; each supports
cron
,
interval
, or
rrule
and
timezone
. Docs and references: - Prefect YAML guide: prefect.yaml - Schedules in Prefect 3: Schedules - Storing/pulling flow code: Store flow code - Docker step functions: prefect-docker steps If you share your target registry/image naming and whether your workers will clone code or run from image-only, I can tailor the template exactly to your setup.