@Marvin with Prefect 2 is there a way to use environment variables in the pull section, I want the env var to be substituted at deploy time i.e. when running

prefect --no-prompt deploy  -n data

- prefect.deployments.steps.git_clone:
    repository: <https://gitlab.com/vee-engineering/services/irs-data.git>
    branch: '{{ env.DEPLOYMENT_BRANCH }}'
    access_token: '{{ prefect.blocks.secret.gitlab-prefect-pipelines}}'

I think this should just work if you swap

'{{ env.DEPLOYMENT_BRANCH }}'

for

'{{ $DEPLOYMENT_BRANCH }}'

In Prefect 2.x, you can indeed use environment variables at deployment time by incorporating them into your deployment configuration. The documentation suggests that although not specifically detailed, environment variables can be set using a

job_variables

section in the

prefect.yaml

file. Here's a guide on how you can structure your deployment file with environment variables: You might want to define your deployment configuration similar to this format:

yaml
deployments:
- name: your-deployment
  entrypoint: your_project/your_flow.py:flow_function
  work_pool:
    name: your_work_pool
  pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://your-repo-url.git>
      branch: '{{ env.DEPLOYMENT_BRANCH }}'
      access_token: '{{ prefect.blocks.secret.gitlab-prefect-pipelines }}'
  # Add job variables here
  job_variables:
    env:
      DEPLOYMENT_BRANCH: "your-default-branch"

This way, you use

env.DEPLOYMENT_BRANCH

as an environment variable which can be substituted at deploy-time. Make sure that your environment is configured to recognize these environment variables when deploying. You can find more details in the Prefect documentation here: Prefect Deployment Guide.

ah thanks Nate. Was getting some weird results with shell scripts when searching

To use environment variables in the

pull

section for deployment with Prefect 2.x, you need to follow a specific approach as environment variables are not expanded by default. Here’s a way to handle environment variables during deployment: 1. Global Job Variables in Deployment: You can add environment variables through the

job_variables

section in your

prefect.yaml

file. Although this typically applies to job execution rather than deployment, it's the closest Prefect's documentation provides in terms of handling environment variables in deployments. Example:

deployments:
   - name: demo-deployment
     entrypoint: demo_project/demo_flow.py:some_work
     work_pool:
       name: local
     job_variables:
       env:
         DEPLOYMENT_BRANCH: main  # Replace with your environment variable

2. Expanding Environment Variables in Scripts: If you're running a shell script as part of a deployment or job, you need to set

expand_env_vars: true

to expand environment variables. Example:

- prefect.deployments.steps.run_shell_script:
       id: get-user
       script: echo $USER
       stream_output: true
       expand_env_vars: true

Unfortunately, the direct substitution of environment variables in the

git_clone

step, as you have written, isn't directly supported or documented by Prefect for deployment at this stage. You might need to manually resolve them before running the deployment command. For more details, you can refer to these resources: - Overriding Job Variables - Deploying with Prefect No relevant issues were found in GitHub that provide alternative solutions for this specific requirement. If needed, consider filing a GitHub issue for this feature request.

just fyi marvin bot will be invoked each time you edit the message 🙂

Unfortunately, Prefect 2.x does not natively support the direct substitution of environment variables within the

git_clone

step of the deployment YAML file. Here are some approaches you could consider: 1. Pre-Processing the YAML File: Before you run the

prefect deploy

command, use a script to pre-process your deployment YAML file. This script can replace placeholders (e.g.,

{{ env.DEPLOYMENT_BRANCH }}

) with actual environment variable values. 2. Using a Template Engine: Tools like Jinja2 can be used in a pre-deployment step to render your YAML file with environment variables. You can incorporate this into a build process or script that you run before deploying with Prefect. 3. External Configuration Management: Consider using an external configuration management tool to handle your deployment configurations if your deployment process is highly dynamic and frequently requires environment-based adjustments. 4. Custom Prefect Commands: Write a custom script or Prefect command to handle the deployment process, resolving environment variables programmatically and dynamically constructing the deployment setup as required. While these methods require additional setup outside of Prefect, they allow the flexibility needed to insert environment variables into deployment configurations. If you have a workflow that fits into any CI/CD pipeline, integrating it there could also be beneficial. Keep in mind, Prefect's documentation or updates might introduce more native support in the future, so staying up-to-date with their latest releases can be helpful.

Confirmed that simple

branch: '{{ $DEPLOYMENT_BRANCH }}'

works. Thanks @Nate