https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> Using Prefect 2, is there a way to sp...
# marvin-in-the-wild
s
@Marvin Using Prefect 2, is there a way to split deployment configuration in multiple YAML files and include them in the final 
prefect.yaml
file? Goal is to tidy the project a little when the number of deployments rises.
m
Currently, Prefect 2.x doesn't support splitting deployment configurations across multiple YAML files directly. However, you can use YAML aliasing and references to tidy up your 
prefect.yaml
file. Here's an example of how you can use a definitions section within your 
prefect.yaml
file to reuse common configurations with aliases and references:
Copy code
yaml
definitions:
  work_pools:
    my_docker_work_pool: &my_docker_work_pool
      name: my-docker-work-pool
      work_queue_name: default
  schedules:
    every_ten_minutes: &every_10_minutes
      interval: 600
  actions:
    docker_build: &docker_build
      - prefect_docker.deployments.steps.build_docker_image: &docker_build_config
          id: build-image
          requires: prefect-docker>=0.3.0
          image_name: my-example-image
          tag: dev
          dockerfile: auto
          push: true

deployments:
  - name: deployment-1
    entrypoint: flows/hello.py:my_flow
    work_pool: *my_docker_work_pool
    schedule: *every_10_minutes
    build: 
      - *docker_build

  - name: deployment-2
    entrypoint: flows/goodbye.py:my_other_flow
    work_pool: *my_docker_work_pool
    build: 
      - <<: *docker_build_config
        dockerfile: Dockerfile.custom
This way, you can keep your deployment configurations more organized and minimize duplication by referencing common configurations from the definitions section. For more details, you can refer to the Prefect 2.x deployment guide here: - Prefect 2.x Deployment Guide
s
@Marvin Using aliases is great but when 50+ deployments are defined, it still results in a pretty big file. Is there anything you can suggest to make it more readable ? Let's say I have 5 different projects with 10 deployments each.
m
Given your need to organize and manage a large number of deployments, one approach could be to separate each project's deployments into individual YAML files and then combine them programmatically before deploying. This way, each YAML file is more manageable, and you can still keep the configurations neat. Here's a step-by-step strategy you can follow: 1. Create separate YAML files for each project: For example, create 
project1.yaml
, 
project2.yaml
, etc., each containing the deployments related to that project.
Copy code
yaml
# project1.yaml
deployments:
  - name: project1-deployment-1
    entrypoint: flows/flow_1.py:flow
    # other config
  - name: project1-deployment-2
    entrypoint: flows/flow_2.py:flow
    # other config
2. Write a script to combine these YAML files: You can use a Python script that reads these individual YAML files and combines them into a single 
prefect.yaml
file. Here's a sample Python script that combines multiple YAML files:
Copy code
python
import yaml
import glob

combined_yaml = {
    'deployments': []
}

for file in glob.glob("projects/*.yaml"):
    with open(file, 'r') as f:
        data = yaml.safe_load(f)
        combined_yaml['deployments'].extend(data.get('deployments', []))

with open("prefect.yaml", 'w') as f:
    yaml.dump(combined_yaml, f)
3. Deploy combined 
prefect.yaml
: Use the Prefect CLI to deploy using the combined file.
Copy code
bash
prefect deploy -p prefect.yaml
By structuring your deployments in separate files and then combining them programmatically, you can manage a large number of deployments more efficiently without cluttering a single large YAML file. For further reading and other configuration options, you can refer to Prefect's documentation: - Prefect 2.x Deployment Guide
3 Views
Open in Slack
PreviousNext
Powered by