@Marvin i have the following prefect.yaml with env variable

GIT_COMMIT_SHA

that I have confirmed is available in my terminal env. prefect.yaml:

deployments:
- name: my-deploy-name
  entrypoint: python/src/.....
  work_pool:
    name: my-cloud-run-pool
    work_queue_name: default
    job_variables:
      image: "<http://gcr.io/universe-big-blue/github.com/universe/big-blue:prefect_{{|gcr.io/universe-big-blue/github.com/universe/big-blue:prefect_{{> $GIT_COMMIT_SHA }}"
      platform: linux/amd64

when i run

prefect deploy --prefect-file python/prefect.yaml

, however, and then try to run the flow via

prefect deployment run

, it seems to replace

GIT_COMMIT_SHA

with an empty string and starts looking for image

<http://gcr.io/blah/blah/blah:prefect_|gcr.io/blah/blah/blah:prefect_>

instead of something like `gcr.io/blah/blah/blah:prefect_fjkdlsaf`where fjkdlsaf is the git commit hash in my env variable. how do i fix this?

It sounds like you're trying to use an environment variable within your

prefect.yaml

to dynamically set the Docker image tag using the

GIT_COMMIT_SHA

environment variable. Prefect's usage of environment variables in configuration files might not directly support the syntax you're using. In general, environment variables in templates or configuration files often need a specific syntax or method to ensure they're correctly parsed and substituted. However, Prefect YAML files themselves do not inherently process environment variables like a shell script would. Here's a way to approach resolving the issue: 1. Pre-process the YAML file: Before calling

prefect deploy

, substitute the environment variable into the YAML file using a tool or script that can replace placeholders with the actual environment variable values. You could use a simple Python script:

python
   import os

   # Load your YAML file
   with open("python/prefect.yaml", "r") as file:
       filedata = file.read()

   # Replace the placeholder with the actual environment variable
   commit_sha = os.getenv("GIT_COMMIT_SHA")
   new_filedata = filedata.replace("{{GIT_COMMIT_SHA}}", commit_sha)

   # Write the replaced content back into the file or another file
   with open("python/prefect.yaml", "w") as file:
       file.write(new_filedata)

2. Run this script before you execute

prefect deploy --prefect-file python/prefect.yaml

. The key here is replacing

{{GIT_COMMIT_SHA}}

with the actual SHA value before Prefect reads the YAML file, as it won't interpret the environment variable directly in the YAML configuration. If you continue to face issues or want a different approach, let me know, and I can further investigate!

@Marvin but doesn't https://github.com/PrefectHQ/prefect/pull/11463/files indicate that we can use custom env vars like

MY_VAR

in the yaml?

It looks like we're on the right track with using environment variables within your

prefect.yaml

file. Prefect supports templating, allowing you to use environment variables in your YAML configuration by employing the double curly bracket syntax

{{ ENV_VAR_NAME }}

or special templating tools designed for this. From the Prefect documentation for both versions 2.x and 3.x, you can indeed use environment variables that are already set in your local environment and reference them in your YAML file using templating: - For Prefect 2.x, it's indicated you can use existing environment variables in deployments. More details can be found here. - In Prefect 3.x, you can use variables in YAML files expressed as strings wrapped in quotes and double curly brackets. You can find more information about customizing deployments using environment variables in Prefect here. In your case, make sure that the templating syntax you are using (

{{ GIT_COMMIT_SHA }}

) matches what's expected by Prefect. You should verify your "prefect.yaml" so it looks something like this using double curly brackets:

yaml
deployments:
- name: my-deploy-name
  entrypoint: python/src/.....
  work_pool:
    name: my-cloud-run-pool
    work_queue_name: default
    job_variables:
      image: "<http://gcr.io/universe-big-blue/github.com/universe/big-blue:prefect_{{> GIT_COMMIT_SHA }}"
      platform: linux/amd64

If you follow this syntax, Prefect should interpolate the environment variable correctly when deploying the flow. If not, check that the

GIT_COMMIT_SHA

environment variable is correctly set and available in the environment where you're running the deployment. Let's hope this kickstarts your deployment process smoothly. Hopefully, this is one of those moments when the universe actually decides to cooperate!