Dot-delimited job-variable env overrides fail silently. @Marvin I have the following at the top of my prefect.yaml

pull:
- prefect.deployments.steps.set_working_directory:
    directory: /work
- prefect.deployments.steps.run_shell_script:
    script: sh scripts/train-branch.sh $GIT_BRANCH
    stream_output: true
    expand_env_vars: true

the env var GIT_BRANCH has a default value of

main

in my work-pool. I am then trying to override it at runtime with:

prefect deployment run -jv env.GIT_BRANCH=my-test-branch run-benchmark/test-adsorption-benchmark

but this override is not working, and

main

is still being used. I also tried removing the env and using

-jv GIT_BRANCH=my-test-branch

, but that didn't work.

It seems there might be a relevant documentation page that discusses how to override job variables for a Prefect deployment, including environment variables. Unfortunately, specific examples showing the exact syntax to override environment variables at runtime aren't given. Therefore, I'll guide you based on the general understanding of Prefect's mechanisms for environment variables. In Prefect 2.x, to override an environment variable for a deployment, you might need to define the override in the

job_variables

section of your

prefect.yaml

file. Alternatively, you could try using a templating approach within the

prefect.yaml

itself. The template syntax would be

{{ $ENV_VAR_NAME }}

, enabling the usage of already-set local environment variables. Here's the relevant information from the Prefect documentation page: - Overriding Job Variables The GitHub issues did not return any results directly addressing the problem with overriding environment variables at runtime with the

-jv

flag. Given the information from the documentation, I'd suggest ensuring that your

prefect.yaml

is correctly formatted and that the

job_variables

section is appropriately set. If you continue to face issues, this might be a candidate for opening a new issue on the Prefect GitHub repository, as this could be a bug or a feature not behaving as expected. Please remember to include the Prefect versions you're using when reporting the issue to the Prefect team.

It seems from the documentation provided by Prefect, particularly from the Overriding Job Variables guide, that you can specify environment variables within your

prefect.yaml

under the

job_variables

section. These should then be available to your deployment. The GitHub issue All environment variables defined in work pools are overridden if any environment variables are defined in a deployment. Improve behavior? #11041 seems to note a related problem, where environment variables defined in one place (such as work pool config) may not behave as expected when other environment variables are provided at deployment time. Unfortunately, the details in the GitHub issues do not offer an immediate solution to the problem you are facing. It might be related to an issue where specifying environment variables during deployment doesn't work correctly with the newer Docker worker concept in Prefect 2.x. To troubleshoot further: 1. Review if you can specify the environment variable in the

prefect.yaml

in the

job_variables

section. 2. Make sure that there aren't conflicting environment variable configurations between your work pool config and your

prefect.yaml

or other deployment methods that might cause precedence issues. 3. Consider filing an issue in the Prefect GitHub repository if the behavior you're experiencing deviates from documentation or expected functionality, as this might be a bug. Given you have tried various methods without success, it seems like more in-depth troubleshooting might be necessary with the Prefect Core team.

FYI: this is a known bug: https://github.com/PrefectHQ/prefect/issues/10931