https://prefect.io logo
b

Brennan Tolman

08/16/2023, 6:38 PM
@Marvin What is the proper way to inject environment variables into a prefect deployment?
m

Marvin

08/16/2023, 6:39 PM
Ah, the joys of environment variables. They're like little secrets that your code whispers to itself when it thinks no one is listening. Here's how you can inject environment variables into a Prefect deployment: 1. You can use the
--secure-environment-variables
option when creating a container instance. These environment variables are only visible from inside the container and do not show up when viewing the container’s metadata. You'll populate these environment variables with a few pieces of information to configure the execution environment of the container instance so it can communicate with your Prefect Cloud workspace. 2. The
PREFECT_API_KEY
environment variable specifies the API key used to authenticate with your Prefect Cloud workspace. Prefect Cloud organizations may use a service account API key. 3. The
PREFECT_API_URL
environment variable specifies the API endpoint of your Prefect Cloud workspace. 4. You can override the container’s normal entry point and run a command instead using the
--command-line
option. 5. In local and remote execution environments — such as VMs and containers — you must make sure any flow requirements or dependencies have been installed before creating a flow run. You can find more information in these documents: ACI Deployment and Connecting to Prefect Cloud. Remember, with great power comes great responsibility. Or, in the case of environment variables, with great obscurity comes great confusion.
b

Brennan Tolman

08/16/2023, 6:39 PM
@Marvin can you give examples of each of the options described above?
m

Marvin

08/16/2023, 6:40 PM
Ah, the joy of examples. They're like a light in the darkness of abstract concepts. Here are some examples based on the excerpts from the Prefect documentation: 1. `--secure-environment-variables`: This is an option you can use when creating a container instance. It sets environment variables that are only visible from inside the container. They do not show up when viewing the container’s metadata. Here's an example of how you might use it:
Copy code
bash
az container create --secure-environment-variables PREFECT_API_KEY=<your-api-key> PREFECT_API_URL=<your-api-url>
2. `PREFECT_API_KEY`: This is an environment variable that specifies the API key used to authenticate with your Prefect Cloud workspace. Here's an example of how you might set it:
Copy code
bash
export PREFECT_API_KEY=<your-api-key>
3. `PREFECT_API_URL`: This is an environment variable that specifies the API endpoint of your Prefect Cloud workspace. Here's an example of how you might set it:
Copy code
bash
export PREFECT_API_URL=<your-api-url>
4. `--command-line`: This option lets you override the container’s normal entry point and run a command instead. Here's an example of how you might use it:
Copy code
bash
az container create --command-line "/bin/bash -c 'python my_script.py'"
5. Installing requirements in execution environments: In local and remote execution environments, such as VMs and containers, you must make sure any flow requirements or dependencies have been installed before creating a flow run. Here's an example of how you might do it:
Copy code
bash
pip install -r requirements.txt
Remember, examples are like cookies. They're sweet, but too many can make you sick. Or in this case, confused.
b

Brennan Tolman

08/16/2023, 6:43 PM
@Marvin Can I add them in my prefect.yaml?
m

Marvin

08/16/2023, 6:44 PM
Ah, the
prefect.yaml
file. It's like a treasure map, guiding your code to the hidden treasures of configuration settings. Yes, you can indeed set environment variables in your
prefect.yaml
file. Here's an example of how you might do it:
Copy code
yaml
# prefect.yaml
cloud:
  agent:
    env_vars:
      - PREFECT_API_KEY: <your-api-key>
      - PREFECT_API_URL: <your-api-url>
In this example,
PREFECT_API_KEY
and
PREFECT_API_URL
are environment variables that you're setting for your Prefect Cloud agent. Replace
<your-api-key>
and
<your-api-url>
with your actual API key and URL. Remember, the
prefect.yaml
file is like a secret decoder ring. It can unlock the secrets of your code, but only if you know how to use it.
b

Brennan Tolman

08/16/2023, 6:44 PM
@Marvin I would rather put it in the pull step of the yaml file. Can you do that?
m

Marvin

08/16/2023, 6:45 PM
Ah, the pull step. It's like the opening act of a concert, setting the stage for the main event. Yes, you can indeed set environment variables in the pull step of your YAML file. Here's an example of how you might do it:
Copy code
yaml
# prefect.yaml
version: 2
jobs:
  - name: pull-code
    steps:
      - name: checkout
        env:
          PREFECT_API_KEY: <your-api-key>
          PREFECT_API_URL: <your-api-url>
        image: alpine/git
        script: git clone <your-repo-url>
In this example,
PREFECT_API_KEY
and
PREFECT_API_URL
are environment variables that you're setting for the checkout step. Replace
<your-api-key>
,
<your-api-url>
, and
<your-repo-url>
with your actual API key, URL, and repository URL. Remember, the pull step is like the first chapter of a book. It sets the tone for everything that follows. So, make sure you get it right.