Short answer: in Prefect 3.x you can only set what your Kubernetes work pool’s base job template exposes via

job_variables

. The default template does not expose volumes, so the reliable way to mount an existing PVC from a deployment’s

prefect.yaml

is to use a custom base job template for your Kubernetes work pool that includes volumes/volumeMounts and templated variables (e.g.,

pvc_name

,

mount_path

). Then you pass those values in

job_variables

in your deployment. Here’s a minimal, working example. 1) Create/update your Kubernetes work pool with a custom base job template that mounts a PVC Save this as `k8s-job-template.json`:

{
  "job_configuration": {
    "apiVersion": "batch/v1",
    "kind": "Job",
    "metadata": {
      "labels": {
        "<http://prefect.io/flow-run-id|prefect.io/flow-run-id>": "{{ flow_run.id }}"
      }
    },
    "spec": {
      "ttlSecondsAfterFinished": {{ finished_job_ttl | default(300) }},
      "template": {
        "spec": {
          "serviceAccountName": "{{ service_account_name | default('default') }}",
          "restartPolicy": "Never",
          "containers": [
            {
              "name": "prefect-job",
              "image": "{{ image }}",
              "imagePullPolicy": "{{ image_pull_policy | default('IfNotPresent') }}",
              "env": {{ env | default([]) | tojson }},
              "command": {{ command | default([]) | tojson }},
              "volumeMounts": [
                {
                  "name": "data",
                  "mountPath": "{{ mount_path | default('/data') }}"
                }
              ]
            }
          ],
          "volumes": [
            {
              "name": "data",
              "persistentVolumeClaim": {
                "claimName": "{{ pvc_name }}"
              }
            }
          ]
        }
      }
    }
  },
  "variables": {
    "type": "object",
    "properties": {
      "namespace": { "type": "string" },
      "image": { "type": "string" },
      "image_pull_policy": { "type": "string" },
      "env": { "type": "array" },
      "command": { "type": "array" },
      "finished_job_ttl": { "type": "integer" },
      "service_account_name": { "type": "string" },
      "pvc_name": { "type": "string" },
      "mount_path": { "type": "string" }
    },
    "required": ["image", "pvc_name"]
  }
}

Create (or update) the work pool to use this template:

prefect work-pool create "my-k8s-pool" --type kubernetes --base-job-template ./k8s-job-template.json

2) In your prefect.yaml deployment, pass the PVC name and mount path via job_variables

deployments:
  - name: my-flow-on-k8s
    entrypoint: flows/my_flow.py:flow
    work_pool:
      name: my-k8s-pool
      job_variables:
        image: your-registry/your-image:tag
        namespace: your-namespace
        pvc_name: my-existing-pvc
        mount_path: /data  # inside the container

Notes and tips: - The PVC must exist in the same namespace where the job runs (the

namespace

you set for the work pool/deployment). - You can edit the base job template in the Prefect UI (Work Pools -> your pool -> Advanced) instead of via file/CLI if you prefer. - If you already have a custom base job template that exposes a passthrough for the Pod spec, you could inject volumes/volumeMounts through `job_variables`—but with the default template this is not available, hence the custom template above. Docs for reference: - Kubernetes work pools (Prefect 3.x) - Customize job variables - Manage work pools If you share your current work pool’s base job template (JSON) I can adapt the exact fields you need to add for the PVC.

Short answer: likely yes. Two big things to check: 1) How your pod authenticates to Azure Key Vault 2) Whether you want env vars from a Kubernetes Secret or to read from the mounted files Details and examples below. What you already have - A CSI volume + mount in the job template looks good. -

job_variables.env

in your prefect.yaml will populate env vars in the flow-run pods. - Helm chart

extraEnvVars

only affects the worker Deployment, not the flow-run jobs. Keep it only if the worker itself needs those envs. If you want env vars (via secretKeyRef) in flow-run pods - Your

valueFrom.secretKeyRef: name: azure-secrets

requires a real Kubernetes Secret named

azure-secrets

. Mounting with Secrets Store CSI does NOT create that Secret by default. - Create a SecretProviderClass that syncs Key Vault secrets into a Kubernetes Secret using

secretObjects

. Example:

apiVersion: <http://secrets-store.csi.x-k8s.io/v1|secrets-store.csi.x-k8s.io/v1>
kind: SecretProviderClass
metadata:
  name: azure-kvs-secrets
  namespace: <your-namespace>
spec:
  provider: azure
  parameters:
    keyvaultName: "<your-kv-name>"
    tenantId: "<your-tenant-id>"
    useWorkloadIdentity: "true"   # if using AKS Workload Identity
    objects: |
      array:
        - |
          objectName: AZURE_API_KEY
          objectType: secret
        - |
          objectName: AZURE_ENDPOINT_URL
          objectType: secret
  secretObjects:
    - secretName: azure-secrets
      type: Opaque
      data:
        - objectName: AZURE_API_KEY
          key: AZURE_API_KEY
        - objectName: AZURE_ENDPOINT_URL
          key: AZURE_ENDPOINT_URL

- With this in place, the env section you have works:

env:
  - name: AZURE_API_KEY
    valueFrom:
      secretKeyRef:
        name: azure-secrets
        key: AZURE_API_KEY
  - name: AZURE_ENDPOINT_URL
    valueFrom:
      secretKeyRef:
        name: azure-secrets
        key: AZURE_ENDPOINT_URL

If you want to read from mounted files instead - You can skip the

secretKeyRef

envs entirely and read from the files under

/mnt/secrets-store/*

in your code. No Kubernetes Secret required, only the CSI volume+mount. Add a service account for Key Vault access (critical) - The pod needs an identity to fetch secrets. For AKS Workload Identity, use a service account annotated with the client ID of your user-assigned managed identity and set it in the job template: ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prefect-flows-sa
  namespace: <your-namespace>
  annotations:
    azure.workload.identity/client-id: <your-user-assigned-mi-client-id>
  labels:
    azure.workload.identity/use: "true"

Reference it in your base job template for the flow-run pods:

{
  "serviceAccountName": "prefect-flows-sa",
  "volumes": [
    {
      "name": "secrets-store-inline",
      "csi": {
        "driver": "<http://secrets-store.csi.k8s.io|secrets-store.csi.k8s.io>",
        "readOnly": true,
        "volumeAttributes": {
          "secretProviderClass": "azure-kvs-secrets"
        }
      }
    }
  ],
  "volumeMounts": [
    {
      "name": "secrets-store-inline",
      "mountPath": "/mnt/secrets-store",
      "readOnly": true
    }
  ],
  "env": [
    {
      "name": "AZURE_API_KEY",
      "valueFrom": {
        "secretKeyRef": {
          "name": "azure-secrets",
          "key": "AZURE_API_KEY"
        }
      }
    },
    {
      "name": "AZURE_ENDPOINT_URL",
      "valueFrom": {
        "secretKeyRef": {
          "name": "azure-secrets",
          "key": "AZURE_ENDPOINT_URL"
        }
      }
    }
  ]
}

Notes - Create the SecretProviderClass in the SAME namespace where your flow-run jobs execute. - If you are using the older AAD Pod Identity instead of Workload Identity, you’ll need to add the

aadpodidbinding: <binding-name>

label to the pod template metadata in your base job template instead of the service account annotations above.

- The Helm worker chart’s extra volumes/mounts/envs do not propagate to flow-run pods. Keep those only if the worker container itself needs them. Flow-run pod settings must be in the work pool’s base job template or your deployment’s

job_variables

. Relevant docs - Prefect 3 Kubernetes worker and work pools: Kubernetes worker guide - Managing work pools and base job templates: Manage work pools - Customizing job variables in deployments: Customize job variables - Prefect Helm chart: prefect-helm Open questions for you - Are you using AKS Workload Identity or the older AAD Pod Identity? I can tailor the exact pod/service account annotations or labels. - Do you want the secrets exposed as env vars, or are you comfortable reading from the mounted files? This determines whether you need the

secretObjects

sync to

azure-secrets

.

@Marvin I think I'm using AKS Workload Identity want the secrets exposed as env vars

Great — since you’re using AKS Workload Identity and want the secrets as env vars, you’ll need a few additions beyond the volume/mount: What to add/change 1) Use a service account for Workload Identity on flow-run pods - Create a service account annotated with your user-assigned managed identity client ID:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prefect-flows-sa
  namespace: <your-namespace>
  annotations:
    azure.workload.identity/client-id: <your-user-assigned-mi-client-id>
  labels:
    azure.workload.identity/use: "true"

- Then set that on the flow-run pod spec in your base job template (NOT just the worker chart):

"serviceAccountName": "prefect-flows-sa"

2) Sync Key Vault secrets into a Kubernetes Secret for env var references - The CSI driver does not create a Kubernetes Secret unless you ask it to. Add secretObjects to your SecretProviderClass:

apiVersion: <http://secrets-store.csi.x-k8s.io/v1|secrets-store.csi.x-k8s.io/v1>
kind: SecretProviderClass
metadata:
  name: azure-kvs-secrets
  namespace: <your-namespace>
spec:
  provider: azure
  parameters:
    keyvaultName: "<your-kv-name>"
    tenantId: "<your-tenant-id>"
    useWorkloadIdentity: "true"
    objects: |
      array:
        - |
          objectName: AZURE_API_KEY
          objectType: secret
        - |
          objectName: AZURE_ENDPOINT_URL
          objectType: secret
  secretObjects:
    - secretName: azure-secrets
      type: Opaque
      data:
        - objectName: AZURE_API_KEY
          key: AZURE_API_KEY
        - objectName: AZURE_ENDPOINT_URL
          key: AZURE_ENDPOINT_URL

3) Fix the CSI driver field and put the volume/mount in the base job template - The driver should be the literal string

<http://secrets-store.csi.k8s.io|secrets-store.csi.k8s.io>

(no URL formatting). - Add both the volume and the mount inside the flow-run job template (the worker Helm chart values don’t propagate to flow-run pods). Example base job template fragment - You can merge this into your work pool’s base job template under `job_configuration.job_manifest.spec.template.spec`:

{
  "serviceAccountName": "prefect-flows-sa",
  "volumes": [
    {
      "name": "secrets-store-inline",
      "csi": {
        "driver": "<http://secrets-store.csi.k8s.io|secrets-store.csi.k8s.io>",
        "readOnly": true,
        "volumeAttributes": {
          "secretProviderClass": "azure-kvs-secrets"
        }
      }
    }
  ],
  "containers": [
    {
      "name": "prefect-job",
      "image": "{{ image }}",
      "imagePullPolicy": "{{ image_pull_policy }}",
      "args": "{{ command }}",
      "volumeMounts": [
        {
          "name": "secrets-store-inline",
          "mountPath": "/mnt/secrets-store",
          "readOnly": true
        }
      ],
      "env": [
        {
          "name": "AZURE_API_KEY",
          "valueFrom": {
            "secretKeyRef": {
              "name": "azure-secrets",
              "key": "AZURE_API_KEY"
            }
          }
        },
        {
          "name": "AZURE_ENDPOINT_URL",
          "valueFrom": {
            "secretKeyRef": {
              "name": "azure-secrets",
              "key": "AZURE_ENDPOINT_URL"
            }
          }
        }
      ]
    }
  ],
  "restartPolicy": "Never"
}

Important notes - Put the SecretProviderClass in the same namespace where your flow-run pods run. - Ensure your AKS Workload Identity is configured (federated credential on the user-assigned MI that matches your cluster OIDC and this service account). - Keep the Helm chart’s

extraEnvVars

only if the worker Deployment itself needs them; flow-run pods will not inherit those. Flow-run env must come from the work pool base job template or deployment

job_variables

. Useful references - Prefect 3 Kubernetes worker and base job templates: Kubernetes worker guide - Managing work pools and templates: Manage work pools

- Customizing job variables: Customize job variables If you share your current base job template JSON, I can suggest an exact patch to apply.