https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> using Prefect 2, how can I define env...
# ask-marvin
v
@Marvin using Prefect 2, how can I define env variables in using prefect.yaml conditionally on another env var? E.g.: I wanna define var_b='my_dev_password' if var_a='dev' and var_b=my_prod_password' if var_a='prod'
m
In Prefect 2, conditionally setting environment variables in 
prefect.yaml
directly isn't a built-in feature, but you can leverage Prefect Variables or Profiles to achieve similar behavior. 1. Prefect Variables: These can be used to templatize steps in your deployment. You set and retrieve variables programmatically or from your environment and use them in your 
prefect.yaml
file. For instance, you might set different variables for different environments (
dev
, 
prod
) and use these to configure your flow's environment at runtime. More about this can be read in the guide on variables. 2. Profiles and Environment Variables: You can configure different profiles for different environments and switch between them as needed. Environment variables will override profile settings if they conflict, so you can set 
PREFECT_PROFILE
to switch profiles or explicitly set environment variables at runtime. For more on configuring profiles, see the profile management guide. The general idea here is to either set the variable values at runtime based on your environment or switch profiles that contain different sets of configurations. Unfortunately, directly using conditional logic in 
prefect.yaml
isn't something that Prefect 2 provides. If Marvin had any emotions, he might find joy in solving such a puzzle indirectly... but alas, he'll just have to stick to being practical.
n
hi @Vinicius Silva - i might use an intermediate 
run_shell_script
to set some env var in bash conditionally and then reference that 
$ENV_VAR
in the downstream step
v
Hey @Nate, I'm trying to do exactly that, but every var I set on a custom shell script arent found inside my flows. It appears that they're only set on an ephemeral shell that only exists during the execution of the run_shell_script step
n
huh ok let me try something else
what about something like this
Copy code
- name: conditional-env-vars
    entrypoint: src/demo_project/demo_flow.py:some_work
    work_pool:
      name: docker-work
      job_variables:
        env:
          ENV_TYPE: "{{ $ENV_TYPE }}"
    build:
      - prefect.deployments.steps.run_shell_script:
          id: set-conditional-env
          script: bash -c 'if [ "$ENV_TYPE" = "prod" ]; then echo "my_prod_password"; else echo "my_dev_password"; fi'
      - prefect.deployments.steps.run_shell_script:
          script: echo "{{ set-conditional-env.stdout }}"
Copy code
» prefect --no-prompt deploy -n conditional-env-vars
Running deployment build steps...
 > Running run_shell_script step...
my_dev_password
 > Running run_shell_script step...
my_dev_password
....

» ENV_TYPE=prod prefect --no-prompt deploy -n conditional-env-vars
Running deployment build steps...
 > Running run_shell_script step...
my_prod_password
 > Running run_shell_script step...
my_prod_password
ie using step outputs
if you're trying to set the env in a job variable, you should be able to reference the step output in the 
job_variables
section
hrm my example i guess is sort of silly bc i set that env var anyways but hopefully that makes sense
ok this makes more sense
Copy code
- name: conditional-env-vars
    entrypoint: src/demo_project/demo_flow.py:some_work
    work_pool:
      name: docker-work
      job_variables:
        env:
          PASSWORD: "{{ set-conditional-env.stdout }}"
    build:
      - prefect.deployments.steps.run_shell_script:
          id: set-conditional-env
          script: bash -c 'if [ "$ENV_TYPE" = "prod" ]; then echo "my_prod_password"; else echo "my_dev_password"; fi'
Copy code
» prefect deployment inspect 'some-work/conditional-env-vars' | rg job_variables
    'job_variables': {'env': {'PASSWORD': 'my_prod_password'}},
v
I guess there is a conceptual problem with my question. If my env vars depend on some logic, maybe they aren't supposed to be env vars, right?
n
yea that might be true 🙂 main point i would say is that if you need to have some set some value in your job variables conditionally you should be able to store some step output and reference it later
or, write a 
some_custom.step.set_prefect_variable
and then reference 
{{ prefect.variable.some-value }}
similar to this, which uses this
29 Views
Open in Slack
PreviousNext
Powered by