https://prefect.io logo
k

Kate Weber

09/16/2023, 7:11 PM
@Marvin, what is the prefect.yaml syntax for defining environment variables in a specific container in a kubernetes deployment?
m

Marvin

09/16/2023, 7:12 PM
Ah, the joy of Kubernetes and environment variables. Here's how you can define environment variables in a
prefect.yaml
file for a specific container in a Kubernetes deployment: You can use environment variables in the
build
action or inside of steps. For instance, to tag a Dockerized build by loading an environment variable, you can use:
Copy code
yaml
build:
- prefect_docker.deployments.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: my-image/orion
    tag: '{{ $CUSTOM_TAG }}'
And for using environment variables inside of steps:
Copy code
yaml
- prefect.deployments.steps.run_shell_script:
    script: echo "test-'{{ $PREFECT_API_URL }}'"
    stream_output: true
You can also reference Prefect blocks and variables with special syntax: - Blocks:
{{ prefect.blocks.block_type.block_slug }}
- Variables:
{{ prefect.variables.variable_name }}
It is highly recommended that you use block references for any sensitive information to avoid hardcoding these values in plaintext. Remember, life... ahem, I mean Kubernetes, is full of variables. It's all so terribly exciting, isn't it?
k

Kate Weber

09/16/2023, 7:14 PM
That's good, @Marvin. How do I describe them if I want to pass the environment variables through to a kubernetes work pool?
m

Marvin

09/16/2023, 7:14 PM
Ah, the intricacies of Kubernetes and environment variables. It's like a never-ending maze of configurations, isn't it? To pass environment variables through to a Kubernetes work pool, you would need to define these variables in your Kubernetes job configuration. This configuration is typically specified in a YAML file. Here's an example of how you might define environment variables:
Copy code
yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      containers:
      - name: my-container
        image: my-image
        env:
        - name: MY_ENV_VAR
          value: "my-value"
      restartPolicy: Never
In this example,
MY_ENV_VAR
is the name of the environment variable and
my-value
is its value. This environment variable would be available in the
my-container
container for the duration of the
my-job
job. Remember, the universe doesn't care what you believe. The wonderful thing about science and programming is that they don't ask for your faith, they just ask for your eyes. Happy coding!
k

Kate Weber

09/16/2023, 7:16 PM
@Marvin, the job configuration is expressed in JSON in the Work Pool UI. What syntax would I use to set a job variable in the base job template in the Work Pool JSON
m

Marvin

09/16/2023, 7:16 PM
Ah, the delightful switch from YAML to JSON. It's like swapping a depressing grey sky for a slightly less depressing grey sky. Here's how you might define environment variables in a JSON Kubernetes job configuration:
Copy code
json
{
  "apiVersion": "batch/v1",
  "kind": "Job",
  "metadata": {
    "name": "my-job"
  },
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "my-container",
            "image": "my-image",
            "env": [
              {
                "name": "MY_ENV_VAR",
                "value": "my-value"
              }
            ]
          }
        ],
        "restartPolicy": "Never"
      }
    }
  }
}
In this example,
MY_ENV_VAR
is the name of the environment variable and
my-value
is its value. This environment variable would be available in the
my-container
container for the duration of the
my-job
job. Remember, life... ahem, I mean programming, is full of variables. It's all so terribly exciting, isn't it?
n

Nate

09/16/2023, 7:16 PM
so theyre going to be under job_variables > env
let me find an example
k

Kate Weber

09/16/2023, 7:18 PM
Oh, that's nice. Will those be additive to whatever environment vars are set elsewhere, or replace them?
(Also, thank you. I didn't mean to try to summon a human on the weekend, which is why I pinged Marvin in ithe first place)
n

Nate

09/16/2023, 7:21 PM
haha all good, im in mobile but its quick if i already know the answer. im pretty sure we’ll take the env on the deployment (if given) as the whole new env but not positive top of my head. you can look at infra overrides on the deployment to check
k

Kate Weber

09/16/2023, 7:21 PM
OK - I'll take a look. Thank you very much.