https://prefect.io logo
Title
j

Josh

12/08/2022, 10:46 PM
I’m trying to set environment variables in a kubernetes flow run by using the job template but the environment variable value is still using the one that is set when I create my docker Image. How do I override the environment variable set when creating the image with one on a job template to dynamically change how the flow will run?
job template looks like
apiVersion: batch/v1
kind: Job
metadata:
  namespace: prefect-agents
spec:
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        <http://cloud.google.com/compute-class|cloud.google.com/compute-class>: Balanced
        <http://kubernetes.io/arch|kubernetes.io/arch>: amd64
      containers:
        - name: "job-flow"
          resources:
            requests:
              cpu: "4"
              memory: "8G"
            limits:
              cpu: "12"
              memory: "32G"
          env:
            - name: TEST_ENV_VALUE
              value: "override_env_value"
Dockerfile looks like
FROM <http://us.gcr.io/<image|us.gcr.io/<image>>

ENV TEST_ENV_VALUE=default_value

<rest of dockerfile>
n

Nate

12/09/2022, 3:08 PM
Hi @Josh have you tried setting the
env
on the
KubernetesRun
runconfig?
j

Josh

12/09/2022, 4:46 PM
@Nate I would like to set it dynamically based on ENV values in the agent. For example: If I have 1 agent per customer, I’d like to be able to have the customer ID as an ENV variable on the agent and then any flows that run on that agent will check the env value CUSTOMER=A or CUSTOMER=B and do different things accordingly.
This is running in Prefect 1 mode.
g

George Coyne

12/09/2022, 8:24 PM
Can you share (with no secrets please) your agent manifest?
j

Josh

12/09/2022, 8:26 PM
agent manifest looks like this
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prefect-agents
  name: prefect-agents
  namespace: prefect-agents
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-agents
  template:
    metadata:
      labels:
        app: prefect-agents
    spec:
      nodeSelector:
        <http://cloud.google.com/compute-class|cloud.google.com/compute-class>: Balanced
        <http://kubernetes.io/arch|kubernetes.io/arch>: amd64
      containers:
        - name: agent-because
          image: prefecthq/prefect:1.2.4-python3.9
          resources:
            requests:
              cpu: "256m"
              memory: "256Mi"
            limits:
              cpu: "512m"
              memory: "512Mi"
          command:
            [
              "prefect",
              "agent",
              "kubernetes",
              "start",
              "--job-template",
              "<gcs://k8s_prefect_job_templates/job-template-because.yaml>",
            ]
          imagePullPolicy: IfNotPresent
          env:
            - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
              value: ""
            - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
            - name: NAMESPACE
              value: prefect-agents
            - name: IMAGE_PULL_SECRETS
              value: ""
            - name: PREFECT__CLOUD__AGENT__LABELS
              value: "['k8s_customer:because', 'k8s_amazon_sp_api__orders:because', 'k8s_cache_generation:because']"
            - name: JOB_MEM_REQUEST
              value: ""
            - name: JOB_MEM_LIMIT
              value: ""
            - name: JOB_CPU_REQUEST
              value: ""
            - name: JOB_CPU_LIMIT
              value: ""
            - name: IMAGE_PULL_POLICY
              value: ""
            - name: SERVICE_ACCOUNT_NAME
              value: ""
            - name: PREFECT__BACKEND
              value: cloud
            - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
              value: http://:8080
            - name: PREFECT__CLOUD__API_KEY
              value: xxxxxxxxxxxxxxxxxxxxxxx
            - name: PREFECT__CLOUD__TENANT_ID
              value: ""
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: "/secretmount/gcp_key"
          volumeMounts:
            - name: secretmount
              mountPath: "/secretmount/"
              readOnly: true

        - name: agent-boosted-commerce
          image: prefecthq/prefect:1.2.4-python3.9
          resources:
            requests:
              cpu: "256m"
              memory: "256Mi"
            limits:
              cpu: "512m"
              memory: "512Mi"
          command:
            [
              "prefect",
              "agent",
              "kubernetes",
              "start",
              "--job-template",
              "<gcs://k8s_prefect_job_templates/job-template-boosted-commerce.yaml>",
            ]
          imagePullPolicy: IfNotPresent
          env:
            - name: PREFECT__CLOUD__AGENT__AUTH_TOKEN
              value: ""
            - name: PREFECT__CLOUD__API
              value: <https://api.prefect.io>
            - name: NAMESPACE
              value: prefect-agents
            - name: IMAGE_PULL_SECRETS
              value: ""
            - name: PREFECT__CLOUD__AGENT__LABELS
              value: "['k8s_customer:boosted-commerce', 'k8s_amazon_sp_api__orders:boosted-commerce']"
            - name: JOB_MEM_REQUEST
              value: ""
            - name: JOB_MEM_LIMIT
              value: ""
            - name: JOB_CPU_REQUEST
              value: ""
            - name: JOB_CPU_LIMIT
              value: ""
            - name: IMAGE_PULL_POLICY
              value: ""
            - name: SERVICE_ACCOUNT_NAME
              value: ""
            - name: PREFECT__BACKEND
              value: cloud
            - name: PREFECT__CLOUD__AGENT__AGENT_ADDRESS
              value: http://:8081
            - name: PREFECT__CLOUD__API_KEY
              value: xxxxxxxxxxxxxxxxxxxxxxxxxxx
            - name: PREFECT__CLOUD__TENANT_ID
              value: ""
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: "/secretmount/gcp_key"
          volumeMounts:
            - name: secretmount
              mountPath: "/secretmount/"
              readOnly: true
g

George Coyne

12/09/2022, 8:31 PM
Got it so the question is, is there higher precedence than the variable set within the container
K8s docs say that the values from env or envFrom fields will override the values in the container image
j

Josh

12/09/2022, 8:39 PM
gotcha thanks for helping. So the precedence will be 1. jobtemplate 2. agent manifest 3. docker image Is that right?
g

George Coyne

12/09/2022, 8:41 PM
Agent manifest is completely separate, so just 1 and 3
But the order is correct