Yes. In Prefect 3.x with the Kubernetes work pool/worker, you can set environment variables in

job_variables

that reference Kubernetes Secrets using

valueFrom.secretKeyRef

. Key points: - Use the list format for

env

(not a simple dict) when referencing secrets. - You can set this at either the work pool level (base job template) or per-deployment via

job_variables

. - Ensure the Secret exists in the same namespace where the job runs. Example in a deployment (

prefect.yaml

):

deployments:
  - name: my-deployment
    work_pool:
      name: kubernetes
      job_variables:
        env:
          - name: DATABASE_PASSWORD
            valueFrom:
              secretKeyRef:
                name: db-credentials
                key: password
          - name: API_TOKEN
            valueFrom:
              secretKeyRef:
                name: api-secrets
                key: token

Programmatic deploy:

from prefect import flow

@flow
def my_flow():
    ...

my_flow.from_source("<https://github.com/owner/repo>").deploy(
    name="my-deployment",
    work_pool_name="kubernetes",
    job_variables={
        "env": [
            {
                "name": "DATABASE_PASSWORD",
                "valueFrom": {
                    "secretKeyRef": {
                        "name": "db-credentials",
                        "key": "password",
                    }
                }
            }
        ]
    },
)

Work pool base job template (UI → Work Pools → your pool → Advanced → Job configuration):

{
  "job_configuration": {
    "env": [
      {
        "name": "APP_ENV",
        "value": "production"
      },
      {
        "name": "DB_PASSWORD",
        "valueFrom": {
          "secretKeyRef": {
            "name": "postgres-secret",
            "key": "password"
          }
        }
      }
    ]
  }
}

Notes: - Use the same namespace for your Secret and your Prefect job. - For more complex pod-level customizations, you can also use the job configuration’s pod overrides, but

env

with

valueFrom.secretKeyRef

is the simplest path. If you share whether you prefer setting this at the work pool or deployment level, I can tailor an example to your setup. For reference docs, see: - Workers and work pools - prefect-kubernetes integration docs

Thanks for the error details — that 409 is coming from server-side validation of

job_variables

against the Kubernetes work pool’s variables schema. What’s happening: - The Kubernetes worker’s job configuration does support

env

as a list (so you can use

valueFrom.secretKeyRef

). - But the server validates deployment

job_variables

against the work pool’s variables schema, where

env

is defined as a simple mapping of string->string. Because of that, passing a list to

job_variables.env

gets rejected with “is not valid under any of the given schemas.” How to do it instead (works reliably): 1) Put secret-backed env vars in the work pool’s base job template - UI: Work Pools → your K8s pool → Advanced → Job configuration - Add

env

as a list so you can use

valueFrom.secretKeyRef

{
  "job_configuration": {
    "env": [
      {
        "name": "DATABASE_PASSWORD",
        "valueFrom": {
          "secretKeyRef": {
            "name": "db-credentials",
            "key": "password"
          }
        }
      },
      {
        "name": "API_TOKEN",
        "valueFrom": {
          "secretKeyRef": {
            "name": "api-secrets",
            "key": "token"
          }
        }
      }
    ]
  }
}

- This applies to all deployments using that pool. If you need different secrets per deployment, create separate K8s work pools with different base job templates. 2) Or set it directly in the job manifest (also in the work pool base job template) - You can edit

job_configuration.job_manifest

and put the env on the container:

{
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "env": [
                  {
                    "name": "DATABASE_PASSWORD",
                    "valueFrom": {
                      "secretKeyRef": {
                        "name": "db-credentials",
                        "key": "password"
                      }
                    }
                  }
                ]
              }
            ]
          }
        }
      }
    }
  }
}

Notes: - Deployment-level

job_variables.env

must be a dict (simple key/value). It cannot carry

valueFrom

structures. - Ensure your K8s Secret is in the same namespace your Prefect jobs run in. - If you need per-deployment variability from code, your options are: - Create separate work pools with different base templates - Use Prefect Variables/Secrets for simple values (not K8s secret refs) Docs: - Work pools overview: https://docs.prefect.io/latest/concepts/work-pools - Kubernetes integration: https://prefecthq.github.io/prefect-kubernetes/ If you share whether you prefer the base template env or the manifest approach, I can provide a tailored JSON you can paste into your pool.

Great data point. Here’s what’s going on and how to make sure deployment-level env vars are not ignored when your work pool base template already sets

env

as a list (with

valueFrom

). What happens under the hood: - During deployment creation,

job_variables.env

is validated against the pool’s variables schema. That schema expects a dict of key->value, not a list. So for deployments you should pass

env

as a dict (e.g.,

{"FOO": "bar"}

), not a list. - At runtime, the Kubernetes worker converts the base template’s list-format env into a dict (only the entries with simple

value

), merges in the deployment

env

dict, then converts back to a list and re-attaches any

valueFrom

entries. Finally it writes the merged list into the Job manifest. So deployment env should not be ignored, even when the pool uses list-format with secrets. Why you might see “ignored” behavior: - If the container’s

env

is hardcoded in the job manifest template and uses the same names, the worker de-duplicates and your deployment vars may appear missing if the template values override them. The worker’s merge logic keeps deployment/work-pool vars first, then appends hardcoded template env that are not duplicates. If the template already sets the same

name

, that template item will be dropped in favor of the merged vars. - If you’re passing deployment env as a list, it gets rejected earlier (or not merged) — always use dict for deployment

job_variables.env

. - If

env

in the pool template is only present under

job_configuration.job_manifest.spec.template.spec.containers[0].env

and you do not also set

job_configuration.env

, the worker still merges, but verify there isn’t a downstream override in your manifest that resets

env

. Checklist to ensure it works: - In the deployment call, pass

job_variables.env

as a dict:

my_flow.from_source("...").deploy(
  name="x",
  work_pool_name="kubernetes",
  job_variables={
    "env": {
      "EXTRA_ONE": "1",
      "EXTRA_TWO": "2"
    }
  }
)

- In the work pool base job template: - Put your secret refs under

job_configuration.env

as a list with

valueFrom

(as you have). - Avoid duplicating the same env var names under

job_configuration.job_manifest.spec.template.spec.containers[0].env

. If you do hardcode env there, ensure names don’t collide with the ones you expect to come from deployment/job_variables. - Confirm the final manifest on a run: - Open a flow run started by the K8s worker and check the created Job/Pod: -

kubectl -n <ns> get pod <name> -o yaml | yq '.spec.containers[0].env'

- You should see: - your deployment dict vars - your work pool list vars (including secret refs) - Prefect-injected vars If you’re still seeing deployment env missing, can you share: - A redacted snippet of your work pool’s “Job configuration” JSON (both

job_configuration.env

and any

job_configuration.job_manifest

env entries) - The exact

job_variables

you’re passing to

.deploy(...)

I can spot the collision/override, or provide a corrected snippet you can paste into the pool.

Thanks for the snippets — that setup should work, and deployment env should not be ignored. What should happen with your config: - Work pool base template has

job_configuration.env

as a list with a secret ref for

DATABASE_PASSWORD

. - Deployment passes

job_variables.env

as a dict with

DATABASE_PASSWORD2: "test"

. At runtime, the K8s worker: - Extracts the list env from the pool, preserves the

valueFrom

entry, converts the simple ones to a dict - Merges in your deployment dict

{"DATABASE_PASSWORD2": "test"}

- Converts back to a list and re-attaches the

valueFrom

secret entry - Writes the merged env into the Job manifest You should see both: - DATABASE_PASSWORD (valueFrom secret) - DATABASE_PASSWORD2 = "test" Common reasons it might look ignored: - The base job manifest also hardcodes

containers[0].env

and collisions or ordering hide what you expect. The worker de-duplicates by name, keeping the merged set first and dropping duplicates from the hardcoded template. - The Secret namespace or resource inspection shows an older Pod (from a previous run) instead of the latest one. Quick checks: - Inspect the actual Pod created for your latest flow run: - kubectl -n prefect get pods - kubectl -n prefect get pod <pod-name> -o yaml | yq '.spec.containers[0].env' - Ensure your base job template does not also set the same env names under

job_configuration.job_manifest.spec.template.spec.containers[0].env

. If it still doesn’t appear: - Please share the relevant chunk of your work pool “Job configuration” JSON, including any

job_configuration.job_manifest.spec.template.spec.containers[0].env

entries if present. - Confirm the exact

job_variables

you pass (you shared it above, but include any other keys if present). I can then pinpoint if a hardcoded manifest env is overriding or if there’s a name collision.

Got it — thanks. With

job_configuration.job_manifest.spec.template.spec.containers[0].env: "{{ env }}"

, the worker will overwrite that placeholder with whatever is in the job configuration’s

env

at runtime. Here’s the catch that explains why your deployment

job_variables.env

(dict) appears ignored: - If the work pool base template sets

job_configuration.env

as a list (to use

valueFrom

for secrets), the deployment’s

job_variables.env

dict is not merged into

job_configuration.env

during deployment creation. - At runtime, the K8s worker converts the list to a dict and merges, but only with what’s already in the job configuration. The deployment dict never made it into

self.env

, so it doesn’t show up. Two ways to fix this; I recommend Option A: A) Put secret env entries directly in the manifest; keep

env

in job_variables as a dict - In your work pool base job template, move the secret to the container’s

env

list in the manifest, and do NOT set

job_configuration.env

as a list. - The worker will then merge deployment

job_variables.env

(dict) with the template env list. Example work pool job configuration:

{
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "env": [
                  {
                    "name": "DATABASE_PASSWORD",
                    "valueFrom": {
                      "secretKeyRef": {
                        "name": "db-credentials",
                        "key": "password"
                      }
                    }
                  }
                ]
              }
            ]
          }
        }
      }
    }
  }
}

Then keep your deployment as:

job_variables={
  "namespace": "prefect",
  "env": {
    "DATABASE_PASSWORD2": "test"
  }
}

Resulting container env will include both: - DATABASE_PASSWORD (secret) - DATABASE_PASSWORD2: test B) Put all env (including deployment-specific) into the work pool’s

job_configuration.env

list - This only works if you don’t need per-deployment env values or you create separate pools per variant. Deployment

job_variables.env

won’t merge when the base uses a list. After applying Option A, re-run and check: - kubectl -n prefect get pod <pod-name> -o yaml | yq '.spec.containers[0].env' If you’d like, paste your current “Job configuration” JSON and I’ll tweak it to the exact structure you need. For reference: - Kubernetes integration docs: https://prefecthq.github.io/prefect-kubernetes/ - Work pools overview: https://docs.prefect.io/latest/concepts/work-pools