https://prefect.io
Join Slack
<@ULVA73B9P> What's the best way to handle environment variables in flows?
# marvin-in-the-wild
j
@Marvin What's the best way to handle environment variables in flows?
m
To handle environment variables in your Prefect flows, here are some methods and recommendations sourced from the Prefect documentation and examples: ### Overriding Job Variables One common pattern is to override job variables in the deployment methods. - Define environment variables in the 
prefect.yaml
file or override them during the 
.deploy()
method call. - These environment variables can be set in the runtime environment of the flow run. For example:
Copy code
yaml
env:
  EXECUTION_ENV: staging
  MY_NOT_SO_SECRET_CONFIG: plumbus
Source: Prefect Documentation: Overriding Work Pool Job Variables ### Using Prefect Profiles You can define a profile with specific environment settings and switch between profiles as needed. - Set the environment variable 
PREFECT_PROFILE
to the name of the desired profile:
Copy code
bash
  export PREFECT_PROFILE=foo
- Alternatively, specify it in the CLI command for one-time usage:
Copy code
bash
  prefect --profile "foo" flow-run ls
Environment variables set this way will take precedence over values in the profile settings. Source: Prefect Documentation: Settings ### Code Example Here is a simple code example on how to call any Python code, including Prefect tasks or flows, with environment variables:
Copy code
python
from prefect import flow

@flow(log_prints=True)
def my_workflow():
    print("I can call any python code here, including prefect tasks or flows")

if __name__ == "__main__":
    my_workflow()  # call it like a normal python function

    # or serve it as a long-lived process
    # my_workflow.serve("my-deployment-name")
You can find more examples and details in the provided code examples. --- These methods should help you manage environment variables effectively in your Prefect flows. Let me know if you need further assistance.
j
@Marvin How do you set environment variables in deployments, using the .deploy() method?
m
When deploying a flow using the 
.deploy()
method, you can set environment variables using the 
job_variables
parameter in your deployment definition. Here’s a step-by-step guide on how to do it: ### Example Using 
job_variables
1. Define Environment Variables in the Deployment: You can add a 
job_variables
section in your deployment definition (usually in your 
prefect.yaml
file or within your script). Here’s an example of what that might look like in your `prefect.yaml`:
Copy code
yaml
   deployments:
   - name: demo-deployment
     entrypoint: demo_project/demo_flow.py:some_work
     work_pool:
       name: local
     job_variables:
       env:
         EXECUTION_ENVIRONMENT: staging
         MY_NOT_SO_SECRET_CONFIG: plumbus
     schedule: null
2. Deploy the Flow: Run the following command to deploy the flow with these job variables:
Copy code
bash
   prefect deploy -n demo-deployment
3. Deploy Using 
.deploy()
Method: Alternatively, you can set these variables directly in your deployment script using the 
.deploy()
method.
Copy code
python
   from prefect.deployments import Deployment
   from demo_project.demo_flow import some_work

   Deployment(
       name="demo-deployment",
       flow_location="demo_project/demo_flow.py",
       work_pool_name="local",
       job_variables={
           "env": {
               "EXECUTION_ENVIRONMENT": "staging",
               "MY_NOT_SO_SECRET_CONFIG": "plumbus"
           }
       }
   ).apply()
### Tips - Templating Existing Environment Variables: If you want to use environment variables that are already set in your local environment, you can template these in your deployment definition as follows:
Copy code
yaml
  job_variables:
    env:
      EXECUTION_ENVIRONMENT: "{{ execution_env }}"
      MY_NOT_SO_SECRET_CONFIG: "{{ not_so_secret_config }}"
Then, set the variables in your local environment before deploying:
Copy code
bash
  export execution_env=staging
  export not_so_secret_config=plumbus
  prefect deploy -n demo-deployment
### Documentation Links - Overriding Work Pool Job Variables - Prefect Deploy Guide These steps should allow you to effectively set and manage environment variables in your Prefect deployments. If you need further assistance, feel free to ask.
44 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by