Joshua Caskie
06/25/2024, 5:59 PMMarvin
06/25/2024, 6:00 PMprefect.yaml
file or override them during the .deploy()
method call.
- These environment variables can be set in the runtime environment of the flow run.
For example:
yaml
env:
EXECUTION_ENV: staging
MY_NOT_SO_SECRET_CONFIG: plumbus
Source: Prefect Documentation: Overriding Work Pool Job Variables
### Using Prefect Profiles
You can define a profile with specific environment settings and switch between profiles as needed.
- Set the environment variable PREFECT_PROFILE
to the name of the desired profile:
bash
export PREFECT_PROFILE=foo
- Alternatively, specify it in the CLI command for one-time usage:
bash
prefect --profile "foo" flow-run ls
Environment variables set this way will take precedence over values in the profile settings.
Source: Prefect Documentation: Settings
### Code Example
Here is a simple code example on how to call any Python code, including Prefect tasks or flows, with environment variables:
python
from prefect import flow
@flow(log_prints=True)
def my_workflow():
print("I can call any python code here, including prefect tasks or flows")
if __name__ == "__main__":
my_workflow() # call it like a normal python function
# or serve it as a long-lived process
# my_workflow.serve("my-deployment-name")
You can find more examples and details in the provided code examples.
---
These methods should help you manage environment variables effectively in your Prefect flows. Let me know if you need further assistance.Joshua Caskie
06/25/2024, 7:39 PMMarvin
06/25/2024, 7:39 PM.deploy()
method, you can set environment variables using the job_variables
parameter in your deployment definition. Here’s a step-by-step guide on how to do it:
### Example Using job_variables
1. Define Environment Variables in the Deployment:
You can add a job_variables
section in your deployment definition (usually in your prefect.yaml
file or within your script).
Here’s an example of what that might look like in your `prefect.yaml`:
yaml
deployments:
- name: demo-deployment
entrypoint: demo_project/demo_flow.py:some_work
work_pool:
name: local
job_variables:
env:
EXECUTION_ENVIRONMENT: staging
MY_NOT_SO_SECRET_CONFIG: plumbus
schedule: null
2. Deploy the Flow:
Run the following command to deploy the flow with these job variables:
bash
prefect deploy -n demo-deployment
3. Deploy Using .deploy()
Method:
Alternatively, you can set these variables directly in your deployment script using the .deploy()
method.
python
from prefect.deployments import Deployment
from demo_project.demo_flow import some_work
Deployment(
name="demo-deployment",
flow_location="demo_project/demo_flow.py",
work_pool_name="local",
job_variables={
"env": {
"EXECUTION_ENVIRONMENT": "staging",
"MY_NOT_SO_SECRET_CONFIG": "plumbus"
}
}
).apply()
### Tips
- Templating Existing Environment Variables:
If you want to use environment variables that are already set in your local environment, you can template these in your deployment definition as follows:
yaml
job_variables:
env:
EXECUTION_ENVIRONMENT: "{{ execution_env }}"
MY_NOT_SO_SECRET_CONFIG: "{{ not_so_secret_config }}"
Then, set the variables in your local environment before deploying:
bash
export execution_env=staging
export not_so_secret_config=plumbus
prefect deploy -n demo-deployment
### Documentation Links
- Overriding Work Pool Job Variables
- Prefect Deploy Guide
These steps should allow you to effectively set and manage environment variables in your Prefect deployments. If you need further assistance, feel free to ask.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by