https://prefect.io logo
Join Slack
Powered by
This message was deleted.
# marvin-in-the-wild
s
This message was deleted.
m
To adjust your YAML file for deploying flows in Prefect 2.x (and prepare for use past September 2024), you should use the 
prefect deploy
command instead of the deprecated 
prefect deployment build
command. The 
prefect deploy
command relies on a 
prefect.yaml
file for deployment configurations. Here's what you need to do: 1. Create a 
prefect.yaml
file in your project root directory: This file will define your deployment configuration. 2. Use the 
prefect deploy
command in your script: This will read the configurations from the 
prefect.yaml
file. Given the functionality in your script, the 
prefect.yaml
file might look something like this:
Copy code
yaml
# prefect.yaml

# generic metadata
prefect-version: 2.x
name: your-flow-name

# preparation steps
build:
  - prefect.deployments.steps.run_shell_script:
      id: get-commit-hash
      script: git rev-parse --short HEAD

# runtime steps
pull:
  - prefect.infrastructure.docker.deployments.pull_docker_image:
      image_name: "{{ docker_build.image_name }}:{{ get-commit-hash.stdout }}"
      tag: latest

# deployment configurations
deployments:
  - name: your-deployment-name
    version: "0.1"
    description: "Deployment description"
    tags: 
      - "environment"
      - "parameters.parent"
      - "parameters.child1"
      - "parameters.child2"
    entrypoint: "path_to_your_flow:flow_function"
    parameters:
      key: value
    infra_overrides:
      image: "{{ docker_build.image_name }}:{{ get-commit-hash.stdout }}"
    work_pool:
      name: work_pool_name
      job_variables: {}
### Adjust your Azure Pipeline Script Once you have the 
prefect.yaml
file set up, modify your Azure pipeline script to use the 
prefect deploy
command. Here is an adjusted version:
Copy code
yaml
- script: |
    prefect cloud login --key $KEY --workspace $(PREFECT_WORKSPACE_NAME)
    prefect deploy --name $(deployname)${{ parameters.deploy_suffix }}.$(environment) \
                   --version "$(Build.DefinitionName)$(Build.BuildNumber)" \
                   --work-queue $(workqueue) \
                   --tag $(environment) \
                   --tag ${{ parameters.parent }} \
                   --tag ${{ parameters.child1 }} \
                   --tag ${{ parameters.child2 }} \
                   $(concurrency_limit_string)
  workingDirectory: $(projectpath)
  displayName: 'Deploy to prefect'
  env:
    KEY: $(PREFECT_API_KEY)
### Resources 1. Prefect deployment documentation 2. Prefect example deployments This approach ensures that your configurations are future-proof, adhering to the updated Prefect guidelines. If you have more specific requirements or issues, feel free to ask!
A 
pipeline.yml
file in Azure Pipeline and a 
prefect.yaml
file for Prefect serve different purposes, but they can work together seamlessly. The 
pipeline.yml
file defines the pipeline execution steps, triggers, jobs, and parameters, while the 
prefect.yaml
file is specific to Prefect, defining how your flows are deployed, managed, and executed. Let's break down the differences and how you can integrate Prefect's configurations: ### 
pipeline.yml
Overview: - Defines pipeline triggers, jobs, and task sequences in Azure Pipelines. - Configurable using parameters that can be passed down to jobs. ### 
prefect.yaml
Overview: - Specifies metadata, build, push, and deployment configurations for Prefect flows. - Used by the 
prefect deploy
command to manage flow deployments. ### Integration: You need to define your Prefect-related flow configurations in a 
prefect.yaml
file and reference this file within your Azure pipeline script to deploy your flow using the 
prefect deploy
command. ### Example: Converting Your Master Template 1. Create a 
prefect.yaml
File: Define the flow deployment configurations. Here's an example:
Copy code
yaml
# prefect.yaml

prefect-version: 2.x
name: your-flow-name

# preparation steps
build:
  - prefect.deployments.steps.run_shell_script:
      id: get-commit-hash
      script: git rev-parse --short HEAD

# deployment configurations
deployments:
  - name: $(deployname)
    version: "0.1"
    description: "Deployment description"
    tags: 
      - "env"
      - "($(parent))"
      - "($(child1))"
      - "($(child2))"
    entrypoint: "path_to_your_flow:flow_function"
    parameters:
      key: value
    infra_overrides:
      image: "$(imagerepo):$(varimage)"
    work_pool:
      name: work_pool_name
      job_variables: {}
2. Adjust Your 
pipeline.yml
Script: Make sure your Azure pipeline script triggers the Prefect 
deploy
command and references your 
prefect.yaml
file.
Copy code
yaml
# pipeline.yml file

variables:
  parent: 'parent'
  child1: 'child1'
  child2: 'child2'
  
trigger:
  branches:
    include:
      - main
      - production
  paths:
    include: # Include!
      - 'branch/to/files/*'

jobs:
- template: ../../../deployfiles/master-template.yml
  parameters:
    parent: ${{ variables.parent }}
    child1: ${{ variables.child1 }}
    child2: ${{ variables.child2 }}
    flowname: "flow_name"
    flowfile: "flowfile"
    image: '4.0'
    cron: '*/30 * * * *' 
    day_or: 'False'
    concurrency_limit: '1'
3. Master Template Script Update:
Copy code
yaml
# master-template.yml

... # other steps

- script: |
    prefect cloud login --key $KEY --workspace $(PREFECT_WORKSPACE_NAME)
    prefect deploy --name $(parameters.flowname)${{ parameters.deploy_suffix }}.$(environment) \
                   --version "$(Build.DefinitionName)$(Build.BuildNumber)" \
                   --work-queue $(workqueue) \
                   --tag $(environment) \
                   --tag ${{ parameters.parent }} \
                   --tag ${{ parameters.child1 }} \
                   --tag ${{ parameters.child2 }} \
                   $(concurrency_limit_string)
  workingDirectory: $(projectpath)
  displayName: 'Deploy to prefect'
  env:
    KEY: $(PREFECT_API_KEY)
### Summary: - Keep your existing 
pipeline.yml
file for Azure Pipelines. - Create a new 
prefect.yaml
file for Prefect-specific flow configurations. - Reference the 
prefect.yaml
file in the 
deploy
command within your Azure pipeline script. This setup will ensure a smooth transition to using the 
prefect deploy
command while maintaining your Azure Pipelines configurations. If you have any further questions or need help with specific parts, feel free to ask!
2 Views
Open in Slack
PreviousNext
Powered by