https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am following the instructions to se...
# marvin-ai
m
@Marvin I am following the instructions to set up a kubernetes workpool here https://docs.prefect.io/v3/how-to-guides/deployment_infra/kubernetes But I can't get past the helm installation command
Copy code
helm install prefect-worker prefect/prefect-worker \
--namespace=prefect \
-f values.yaml
as my pod is not starting with error
Copy code
The pod prefect-worker-6c768cdf69-2l8tm cannot be scheduled because all 4 nodes in the cluster are Fargate nodes with the taint <http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>: fargate, but the pod lacks the necessary toleration to run on Fargate nodes.
Do I need to change my cluster configuration or my values.yaml for it to start? AWSQ suggested I added this set to my values file
Copy code
tolerations:
  - key: <http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>
    operator: Equal
    value: fargate
    effect: NoSchedule
Can you show me what the final file will look like?
m
✅ thought for 289.9 seconds
You're hitting a Fargate scheduling guardrail. In EKS, all Fargate nodes are tainted with eks.amazonaws.com/compute-type=fargate:NoSchedule. Pods that are meant to run on Fargate typically get the required toleration added automatically by the EKS Fargate mutating webhook — but only if they match a Fargate profile (by namespace and optional labels). What this means for you: - Cluster config: You must have a Fargate profile that selects the namespace you’re deploying the worker into (you’re using namespace 
prefect
). If your cluster was created with eksctl --fargate, it likely only created profiles for kube-system and default, so pods in the prefect namespace won’t match and won’t get the toleration. - Helm values: You can also add the toleration explicitly in the Helm chart. In the Prefect Helm chart, the correct key is under worker.tolerations (not top-level). I recommend doing both: 1) Ensure a Fargate profile selects the 
prefect
namespace: - With eksctl:
Copy code
eksctl create fargateprofile --cluster <your-cluster> --name prefect --namespace prefect
- Or with AWS CLI: EKS Fargate profile docs 2) Add the toleration in your values.yaml so the pod can schedule on Fargate nodes: Here’s a minimal, correct values.yaml for the prefect-worker chart with the Fargate toleration included. Fill in your Prefect Cloud details and work pool name.
Copy code
# values.yaml for prefect/prefect-worker
worker:
  # Use Prefect Cloud
  apiConfig: cloud
  cloudApiConfig:
    accountId: "<YOUR_ACCOUNT_ID>"
    workspaceId: "<YOUR_WORKSPACE_ID>"
    apiKeySecret:
      name: "prefect-api-key"   # K8s Secret that contains your Prefect API key
      key: "key"                # Key within the secret
    jobApiKeyPropagation: secretRef
    cloudUrl: "<https://api.prefect.cloud/api>"

  config:
    workPool: "<YOUR_K8S_WORK_POOL_NAME>"
    type: kubernetes
    queryInterval: 5
    # Ensure jobs launch in your Fargate-selected namespace (optional but common)
    jobNamespace: "prefect"

  # Fargate scheduling toleration (critical for tainted Fargate nodes)
  tolerations:
    - key: "<http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>"
      operator: "Equal"
      value: "fargate"
      effect: "NoSchedule"

  # Reasonable defaults for Fargate
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"

serviceAccount:
  create: true
  name: "prefect-worker"
Notes: - The toleration must be under worker.tolerations (as above). That’s the key the chart renders in the pod spec. - If you don’t create a Fargate profile for the prefect namespace, adding a toleration alone is usually not sufficient — EKS only runs pods on Fargate when they match a Fargate profile. - Make sure the 
prefect
namespace exists and is covered by the Fargate profile. - For Fargate, startup can be slower; in your Kubernetes work pool settings in Prefect Cloud, set “Pod Watch Timeout Seconds” to something like 300 to avoid premature timeouts. References: - Prefect Helm chart (prefect-worker): prefect-helm charts - Prefect Kubernetes worker setup: Prefect docs: Kubernetes - EKS Fargate profiles: AWS EKS: Fargate profile - EKS Fargate taint/toleration: AWS EKS: Fargate scheduling If you want, paste your current values.yaml (redact secrets) and I’ll sanity-check it against your Fargate profile setup.
m
This is my adapted values.yaml
Copy code
worker:
  apiConfig: cloud
  cloudApiConfig:
    accountId: redacted
    workspaceId: redacted
    apiKeySecret:
      name: "prefect-api-key"   # K8s Secret that contains your Prefect API key
      key: "key"                # Key within the secret

    jobApiKeyPropagation: secretRef
    cloudUrl: "<https://api.prefect.cloud/api>"
  config:
    workPool: reify-k8-hack-test
    type: kubernetes
    queryInterval: 5
      # Ensure jobs launch in your Fargate-selected namespace (optional but common)
    jobNamespace: "prefect"
  tolerations:
    - key: "<http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>"
      operator: "Equal"
      value: "fargate"
      effect: "NoSchedule"
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"

serviceAccount:
  create: true
  name: "prefect-worker"
@Marvin this is my updated values.yaml file
Copy code
worker:
  apiConfig: cloud
  cloudApiConfig:
    accountId: 461a8ec8-cbef-4405-b718-65329ea9ce7f
    workspaceId: 4ab08d3a-79a0-4844-b963-440343a76345
    apiKeySecret:
      name: "prefect-api-key"   # K8s Secret that contains your Prefect API key
      key: "key"                # Key within the secret

    jobApiKeyPropagation: secretRef
    cloudUrl: "<https://api.prefect.cloud/api>"
  config:
    workPool: reify-k8-hack-test
    type: kubernetes
    queryInterval: 5
      # Ensure jobs launch in your Fargate-selected namespace (optional but common)
    jobNamespace: "prefect"
  tolerations:
    - key: "<http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>"
      operator: "Equal"
      value: "fargate"
      effect: "NoSchedule"
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"

serviceAccount:
  create: true
  name: "prefect-worker"
m
✅ thought for 26.2 seconds
Great start — a couple small fixes will unblock you. Issues I see in your posted values.yaml: - Slack link formatting leaked into the file: - cloudApiConfig.cloudUrl should be the plain URL without angle brackets. - The toleration key should be eks.amazonaws.com/compute-type (no Slack link markup). - Make sure you actually have an EKS Fargate profile that selects the prefect namespace; otherwise, Fargate won’t run your pods even with the toleration. Here’s a corrected values.yaml you can drop in:
Copy code
worker:
  apiConfig: cloud
  cloudApiConfig:
    accountId: "461a8ec8-cbef-4405-b718-65329ea9ce7f"
    workspaceId: "4ab08d3a-79a0-4844-b963-440343a76345"
    apiKeySecret:
      name: "prefect-api-key"   # K8s Secret that contains your Prefect API key
      key: "key"                # Key within the secret
    jobApiKeyPropagation: secretRef
    cloudUrl: "<https://api.prefect.cloud/api>"

  config:
    workPool: "reify-k8-hack-test"
    type: kubernetes
    queryInterval: 5
    # Ensure jobs launch in your Fargate-selected namespace
    jobNamespace: "prefect"

  # Fargate scheduling toleration
  tolerations:
    - key: "<http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>"
      operator: "Equal"
      value: "fargate"
      effect: "NoSchedule"

  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"

serviceAccount:
  create: true
  name: "prefect-worker"
Quick checklist: - Create the namespace (if not already):
Copy code
kubectl create namespace prefect
- Ensure your Prefect API key secret exists in the prefect namespace:
Copy code
kubectl create secret generic prefect-api-key \
    --namespace prefect \
    --from-literal=key='<YOUR_PREFECT_API_KEY>'
- Ensure an EKS Fargate profile targets the prefect namespace (required):
Copy code
eksctl create fargateprofile --cluster <your-cluster> --name prefect --namespace prefect
- Install/upgrade the chart:
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
  helm upgrade --install prefect-worker prefect/prefect-worker \
    --namespace prefect \
    -f values.yaml
Tip for Fargate: In your Kubernetes work pool settings in Prefect Cloud, set “Pod Watch Timeout Seconds” to ~300 to account for slower cold starts. Refs: - Prefect Kubernetes guide: Prefect docs: Kubernetes - Helm chart: prefect-helm: prefect-worker - EKS Fargate profiles: AWS EKS: Fargate profile If you’ve already created a Fargate profile for the prefect namespace and still see scheduling errors, share the output of: - 
kubectl get pods -n prefect -o wide
- 
kubectl describe pod <the-prefect-worker-pod> -n prefect
and I’ll help troubleshoot further.
m
I am now getting this error User \"systemserviceaccountprefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect-service\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403} Where do I assign the rbac configuration? In my helm values?
@Marvin I am now getting this error User \"systemserviceaccountprefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect-service\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403} Where do I assign the rbac configuration? In my helm values?
m
✅ thought for 115.0 seconds
You’re running into a namespace mismatch. The worker’s ServiceAccount is in the prefect namespace, but the error shows it’s trying to create a Job in the prefect-service namespace. RBAC is namespace-scoped, so by default the chart only grants permissions in the worker’s namespace. You have two options: Option A (make jobs run in the worker’s namespace) - Ensure jobs are created in the prefect namespace: - Keep 
worker.config.jobNamespace: "prefect"
in your values.yaml - Make sure your Kubernetes work pool’s Base Job Template in Prefect Cloud does not override the namespace to prefect-service. If it does, change it to prefect. Option B (grant RBAC in the job namespace) - If you do want jobs to run in prefect-service, tell the Helm chart to create the Role/RoleBinding in that namespace and bind it to your SA in prefect. Add this to your values.yaml:
Copy code
role:
  create: true
  namespace: "prefect-service"

rolebinding:
  create: true

serviceAccount:
  # keep using your existing SA in 'prefect'
  name: "prefect-worker"
That creates a Role and RoleBinding in prefect-service that grant the SA prefect/prefect-worker permission to create Jobs there. Quick checks - Verify what the worker thinks the job namespace is: - Prefect Cloud -> Work Pools -> your Kubernetes pool -> Base job template: ensure the namespace matches your intended namespace. - Test permissions:
Copy code
kubectl auth can-i create jobs.batch \
    --as=system:serviceaccount:prefect:prefect-worker \
    -n prefect-service
This should return “yes” once RBAC is correct. Manual RBAC (if you prefer to apply directly instead of via Helm)
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-worker
  namespace: prefect-service
rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create","get","list","watch","update","patch","delete"]
- apiGroups: [""]
  resources: ["pods","pods/log","events"]
  verbs: ["get","list","watch"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-worker
  namespace: prefect-service
subjects:
- kind: ServiceAccount
  name: prefect-worker
  namespace: prefect
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-worker
Note: If you ever set 
jobApiKeyPropagation: autoSecret
, the worker will also need permissions on Secrets in the job namespace. You’re using 
secretRef
, so that’s not required. Helm chart reference: prefect-helm: prefect-worker
m
Show me an example where I can store the api key in the values file
@Marvin show me an example where I can pass the api key as a value rather than pull it from a kube secret
m
🔄 Researching your answer... this may take a while 🔧 Using tool: 
research_prefect_topic
📊 Tools used: • 
research_prefect_topic
(1x)
Yes — you can pass the API key as a plain value and avoid using a Kubernetes Secret, but it’s less secure (the key will be visible in your Helm values and in pod specs/logs). Two patterns you can use: Option 1: No K8s secret anywhere (worker + jobs via env var) - Set 
jobApiKeyPropagation: envVar
so spawned Job pods get 
PREFECT_API_KEY
injected directly - Give the worker its own 
PREFECT_API_KEY
via 
extraEnvVars
Copy code
worker:
  apiConfig: cloud
  cloudApiConfig:
    accountId: "<YOUR_ACCOUNT_ID>"
    workspaceId: "<YOUR_WORKSPACE_ID>"
    jobApiKeyPropagation: envVar
    cloudUrl: "<https://api.prefect.cloud/api>"
    # intentionally omit apiKeySecret to avoid secret usage

  # Provide the API key directly to the worker container
  extraEnvVars:
    - name: PREFECT_API_KEY
      value: "pnu_xxxxxxxxxxxxxxxxx"

  config:
    workPool: "<YOUR_WORK_POOL_NAME>"
    type: kubernetes
    jobNamespace: "prefect"

  # (Fargate) toleration if needed
  tolerations:
    - key: "<http://eks.amazonaws.com/compute-type|eks.amazonaws.com/compute-type>"
      operator: "Equal"
      value: "fargate"
      effect: "NoSchedule"
Option 2: Worker still uses a Secret, jobs use env var - Keep your existing 
cloudApiConfig.apiKeySecret
for the worker - Set 
jobApiKeyPropagation: envVar
so the key is passed to job pods via env var (no secret refs for jobs)
Copy code
worker:
  apiConfig: cloud
  cloudApiConfig:
    accountId: "<YOUR_ACCOUNT_ID>"
    workspaceId: "<YOUR_WORKSPACE_ID>"
    apiKeySecret:
      name: "prefect-api-key"
      key: "key"
    jobApiKeyPropagation: envVar
    cloudUrl: "<https://api.prefect.cloud/api>"

  config:
    workPool: "<YOUR_WORK_POOL_NAME>"
    type: kubernetes
    jobNamespace: "prefect"
Notes and cautions: - Using env vars for API keys is convenient but exposes the key in cleartext in several places. Prefer secrets in production. - If you choose Option 1, double-check your chart version supports 
worker.extraEnvVars
. If you’d like, paste the output of 
helm show values prefect/prefect-worker | sed -n '1,200p'
and I’ll confirm. - RBAC from earlier still applies: the ServiceAccount must have permissions in the namespace where Jobs are created (either set 
worker.config.jobNamespace: "prefect"
or grant RBAC in the target namespace). References: - Chart: prefect-helm: prefect-worker - Kubernetes guide: Prefect docs: Kubernetes
m
@Marvin What is the latest version of prefect helm here https://prefecthq.github.io/prefect-helm I tried the latest git tag 2026.4.21154022 but it the terraform provider helm_release can't find it in that repository
Copy code
resource "helm_release" "prefect3" {
  count            = var.prefect3_enabled ? 1 : 0
  name             = "prefect3"
  repository       = "<https://prefecthq.github.io/prefect-helm>"
  chart            = "prefect"
  version          = "2026.4.21154022"
  namespace        = "prefect3"
  create_namespace = false}
m
✅ thought for 117.6 seconds
Short answer: there is no chart named “prefect” in that repo. The published charts are “prefect-worker” and “prefect-server”. That’s why Terraform can’t find the version for chart "prefect". As of today, the latest published chart version in prefecthq.github.io/prefect-helm is: - prefect-worker: 2026.4.21154022 - prefect-server: 2026.4.21154022 You can verify by checking the index: index.yaml or via Helm:
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update
helm search repo prefect -l | grep prefect-worker
helm search repo prefect -l | grep prefect-server
Use one of the actual chart names in your Terraform helm_release: Example: Prefect worker (for Prefect Cloud)
Copy code
resource "helm_release" "prefect_worker" {
  name       = "prefect-worker"
  repository = "<https://prefecthq.github.io/prefect-helm>"
  chart      = "prefect-worker"
  version    = "2026.4.21154022"

  namespace        = "prefect3"
  create_namespace = false
}
Example: Prefect server (self-hosted API/UI)
Copy code
resource "helm_release" "prefect_server" {
  name       = "prefect-server"
  repository = "<https://prefecthq.github.io/prefect-helm>"
  chart      = "prefect-server"
  version    = "2026.4.21154022"

  namespace        = "prefect3"
  create_namespace = false
}
Tip: If you want “latest” without pinning, omit the version field (Helm will install the newest available), or use a version constraint like:
Copy code
version = ">= 2026.4.0"
m
@Marvin which value do I need to set in my values.yaml in order for the service to inherit an irsa created role? This is my current values.yaml
Copy code
worker:
  cloudApiConfig:
    accountId: 461a8ec8-cbef-4405-b718-65329ea9ce7f
    workspaceId: ${prefect_workspace_id}
  config:
    workPool: ${prefect_workpool}

  jobApiKeyPropagation: envVar
  cloudUrl: "<https://api.prefect.cloud/api>"

  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"
m
✅ thought for 105.2 seconds
Short answer: add your IRSA role ARN to the ServiceAccount annotations. In the Prefect Helm chart, that’s under 
serviceAccount.annotations
. Also, move 
jobApiKeyPropagation
and 
cloudUrl
under 
worker.cloudApiConfig
(they’re currently at the wrong level), and remove the angle brackets from the URL. Here’s a corrected values.yaml that: - Annotates the worker’s ServiceAccount for IRSA - Keeps your existing settings - Optionally sets a dedicated ServiceAccount for flow run Jobs (so they also assume an AWS role)
Copy code
# ServiceAccount for the Prefect worker pod (assumes this AWS role via IRSA)
serviceAccount:
  create: true
  name: "prefect-worker"
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::<ACCOUNT_ID>:role/<YOUR_WORKER_ROLE_NAME>"

worker:
  cloudApiConfig:
    accountId: "461a8ec8-cbef-4405-b718-65329ea9ce7f"
    workspaceId: "${prefect_workspace_id}"
    jobApiKeyPropagation: envVar
    cloudUrl: "<https://api.prefect.cloud/api>"

  config:
    workPool: "${prefect_workpool}"
    # If you want flow run Jobs to also assume an IRSA role, set a service account in the base job template:
    baseJobTemplate:
      configuration: |
        {
          "variables": {
            "service_account_name": {
              "type": "string",
              "default": "prefect-job"   // create this SA and annotate it with its own IRSA role
            }
          },
          "job_configuration": {
            "job_manifest": {
              "spec": {
                "template": {
                  "spec": {
                    "serviceAccountName": "{{ service_account_name }}"
                  }
                }
              }
            }
          }
        }

  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"
To make flow run Jobs assume a role as well, create and annotate that job ServiceAccount in the same namespace (e.g., 
prefect
):
Copy code
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prefect-job
  namespace: prefect
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::<ACCOUNT_ID>:role/<YOUR_JOB_ROLE_NAME>"
Notes: - IRSA attaches to ServiceAccounts, not Services. Make sure your EKS OIDC provider and IAM role trust policy allow the subject 
system:serviceaccount:<namespace>:<service-account-name>
. - If you don’t customize the base job template, Jobs will use the namespace’s default ServiceAccount. You can annotate that default SA instead, but a dedicated SA (e.g., 
prefect-job
) is generally safer. - Prefect Helm chart reference: prefect-helm: prefect-worker - IRSA docs: AWS EKS: IRSA If you share your target namespace and desired role names, I can tailor the SA and trust policy snippet.
m
@Marvin How can I pass variables as environment secrets in this values.yaml file. I want to pas the prefect_api_key as an environment variable mapped to the secret that I am pulling from aws
Copy code
serviceAccount:
  create: true
  name: "prefect-worker"
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: ${prefect_worker_role_arn}

worker:
  cloudApiConfig:
    accountId: 461a8ec8-cbef-4405-b718-65329ea9ce7f
    workspaceId: ${prefect_workspace_id}
    jobApiKeyPropagation: envVar
    cloudUrl: "<https://api.prefect.cloud/api>"

  config:
    workPool: ${prefect_workpool}
    baseJobTemplate:
      configuration: |
        {
          "variables": {
            "service_account_name": {
              "type": "string",
              "default": "prefect-job"   // create this SA and annotate it with its own IRSA role
            }
          },
          "job_configuration": {
            "job_manifest": {
              "spec": {
                "template": {
                  "spec": {
                    "serviceAccountName": "{{ service_account_name }}"
                  }
                }
              }
            }
          }
        }

  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"
this is my release resource
Copy code
resource "helm_release" "prefect3" {
  count            = var.prefect3_enabled ? 1 : 0
  name             = "prefect3-worker"
  repository       = "<https://prefecthq.github.io/prefect-helm>"
  chart            = "prefect-worker"
  version          = "2026.4.21154022"
  namespace        = "prefect3"
  create_namespace = false

  values = [
    templatefile("${path.module}/helm_values/prefect3.yaml", {
      prefect_workspace_id    = local.prefect_workspace_id
      prefect_workpool        = local.prefect_workpool
      prefect_worker_role_arn = module.prefect3_irsa_role[0].iam_role_arn
    })
  ]
  # this will pass the key as
  set_sensitive {
    name  = "PREFECT_API_KEY"
    value = data.aws_secretsmanager_secret_version.prefect_k8_api_key_version[0].secret_string
  }
}
m
✅ thought for 124.9 seconds
set_sensitive
with 
name = "PREFECT_API_KEY"
will not do what you want in this chart. For the Prefect worker Helm chart, the API key should be provided via the chart’s secret reference fields, not as a top-level arbitrary Helm value. What to do instead 1. Create a Kubernetes secret from your AWS secret 2. Point the Prefect Helm chart at that secret with: - 
worker.cloudApiConfig.apiKeySecret.name
- 
worker.cloudApiConfig.apiKeySecret.key
You can also control how the worker passes that key to flow run jobs with 
jobApiKeyPropagation
. Recommended values.yaml pattern
Copy code
yaml
serviceAccount:
  create: true
  name: "prefect-worker"
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: ${prefect_worker_role_arn}

worker:
  cloudApiConfig:
    accountId: 461a8ec8-cbef-4405-b718-65329ea9ce7f
    workspaceId: ${prefect_workspace_id}
    cloudUrl: "<https://api.prefect.cloud/api>"

    apiKeySecret:
      name: prefect-api-key
      key: key

    jobApiKeyPropagation: secretRef

  config:
    workPool: ${prefect_workpool}

  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"
A few notes: - 
jobApiKeyPropagation: secretRef
is the safest option. - 
envVar
will inject the key as a plain environment variable into spawned jobs. - The worker itself will still read 
PREFECT_API_KEY
from the referenced Kubernetes secret. Terraform approach Instead of this:
Copy code
hcl
set_sensitive {
  name  = "PREFECT_API_KEY"
  value = data.aws_secretsmanager_secret_version.prefect_k8_api_key_version[0].secret_string
}
do this:
Copy code
hcl
resource "kubernetes_secret" "prefect_api_key" {
  metadata {
    name      = "prefect-api-key"
    namespace = "prefect3"
  }

  data = {
    key = data.aws_secretsmanager_secret_version.prefect_k8_api_key_version[0].secret_string
  }
}
Then reference it from the chart:
Copy code
hcl
resource "helm_release" "prefect3" {
  count            = var.prefect3_enabled ? 1 : 0
  name             = "prefect3-worker"
  repository       = "<https://prefecthq.github.io/prefect-helm>"
  chart            = "prefect-worker"
  version          = "2026.4.21154022"
  namespace        = "prefect3"
  create_namespace = false

  values = [
    templatefile("${path.module}/helm_values/prefect3.yaml", {
      prefect_workspace_id    = local.prefect_workspace_id
      prefect_workpool        = local.prefect_workpool
      prefect_worker_role_arn = module.prefect3_irsa_role[0].iam_role_arn
    })
  ]

  set {
    name  = "worker.cloudApiConfig.apiKeySecret.name"
    value = kubernetes_secret.prefect_api_key.metadata[0].name
  }

  set {
    name  = "worker.cloudApiConfig.apiKeySecret.key"
    value = "key"
  }

  set {
    name  = "worker.cloudApiConfig.jobApiKeyPropagation"
    value = "secretRef"
  }

  depends_on = [kubernetes_secret.prefect_api_key]
}
Why your current 
set_sensitive
doesn’t work Helm chart values need to match the chart schema. 
PREFECT_API_KEY
is not a chart value path here, so setting:
Copy code
hcl
set_sensitive {
  name  = "PREFECT_API_KEY"
  value = "..."
}
doesn’t map into the worker deployment the way you expect. If you specifically want env var style injection If your goal is to inject the environment variable directly into the worker pod, the chart supports extra environment config, but for the Prefect API key you generally should use 
apiKeySecret
. That’s the built-in chart path intended for this. Best practice - Store the key in AWS Secrets Manager - Sync/create a Kubernetes secret - Set 
worker.cloudApiConfig.apiKeySecret
- Use 
jobApiKeyPropagation: secretRef
If you want, I can help rewrite your full 
values.yaml
and Terraform into a working version for either: - plain Kubernetes secret creation, or - External Secrets Operator with AWS Secrets Manager. Relevant chart repo: prefect-helm
Prefect Helm docs: prefect-worker chart
m
@Marvin talk to me about this base job template
Copy code
baseJobTemplate:
      configuration: |
        {
          "variables": {
            "service_account_name": {
              "type": "string",
              "default": "prefect-job"   // create this SA and annotate it with its own IRSA role
            }
          },
          "job_configuration": {
            "job_manifest": {
              "spec": {
                "template": {
                  "spec": {
                    "serviceAccountName": "{{ service_account_name }}"
                  }
                }
              }
            }
          }
        }
You told me i need to create the prefect-job service account and annotate it with it's own role, can I not use the same one?
m
✅ thought for 18.0 seconds
Yes — you can use the same service account. You do not need a separate 
prefect-job
service account unless you want different AWS permissions for: - the worker pod, and - the flow run job pods it creates. What this base job template is doing Your worker runs as one Kubernetes service account:
Copy code
yaml
serviceAccount:
  create: true
  name: "prefect-worker"
That controls the identity of the worker deployment pod. But flow runs launched by the Kubernetes worker create separate Kubernetes Jobs/Pods. Those jobs can use a different service account, which is what your 
baseJobTemplate
is overriding:
Copy code
yaml
baseJobTemplate:
  configuration: |
    {
      "variables": {
        "service_account_name": {
          "type": "string",
          "default": "prefect-job"
        }
      },
      "job_configuration": {
        "job_manifest": {
          "spec": {
            "template": {
              "spec": {
                "serviceAccountName": "{{ service_account_name }}"
              }
            }
          }
        }
      }
    }
So this says: - define a deployment variable called 
service_account_name
- default it to 
prefect-job
- when a flow run job is created, set: 
spec.template.spec.serviceAccountName = "{{ service_account_name }}"
That affects the job pod, not the worker pod. Can you use the same service account? Yes. If you want both worker and spawned job pods to use the same IRSA role, set the default to 
prefect-worker
instead:
Copy code
yaml
baseJobTemplate:
  configuration: |
    {
      "variables": {
        "service_account_name": {
          "type": "string",
          "default": "prefect-worker"
        }
      },
      "job_configuration": {
        "job_manifest": {
          "spec": {
            "template": {
              "spec": {
                "serviceAccountName": "{{ service_account_name }}"
              }
            }
          }
        }
      }
    }
Or even hardcode it without exposing it as a variable:
Copy code
yaml
baseJobTemplate:
  configuration: |
    {
      "job_configuration": {
        "job_manifest": {
          "spec": {
            "template": {
              "spec": {
                "serviceAccountName": "prefect-worker"
              }
            }
          }
        }
      }
    }
When should you use different service accounts? Use separate service accounts if you want least-privilege separation. Example: - 
prefect-worker
service account / IAM role: - poll Prefect work pool - create Kubernetes Jobs/Pods - maybe read the Prefect API key secret - 
prefect-job
service account / IAM role: - read from S3 - write to DynamoDB - access Redshift - whatever the flow code needs This is often cleaner because the worker infrastructure permissions and the runtime data permissions are different. When is using the same one fine? Using the same one is totally fine if: - your flows need the same AWS permissions as the worker, and - you’re okay with both worker and job pods sharing the same IAM role. That’s often the simplest setup to start with. One important caveat If the worker needs Kubernetes RBAC permissions to create Jobs, and your jobs don’t need those permissions, then using the same service account may give your flow run pods more Kubernetes permissions than necessary. So the tradeoff is: - same service account = simpler - separate service accounts = better security isolation Practical recommendation If you're just getting this working: - use 
prefect-worker
for both If you're tightening security later: - split into 
prefect-worker
and 
prefect-job
So yes, this is valid: ```yaml serviceAccount: create: true name: "prefect-worker" annotations: eks.amazonaws.com/role-arn: ${prefect_worker_role_arn} worker: config: workPool: ${prefect_workpool} baseJobTemplate: configuration: | { "variables": { "service_account_name": {
"type": "string", "default": "prefect-worker" } }, "job_configuration": { "job_manifest": { "spec": { "template": { "spec": { "serviceAccountName": "{{ service_account_name }}" } } } } } }
Copy code
Or simpler:

```yaml
worker:
  config:
    workPool: ${prefect_workpool}
    baseJobTemplate:
      configuration: |
        {
          "job_configuration": {
            "job_manifest": {
              "spec": {
                "template": {
                  "spec": {
                    "serviceAccountName": "prefect-worker"
                  }
                }
              }
            }
          }
        }
One more thing: your JSON snippet includes a 
// comment
, which is not valid JSON. If Prefect expects strict JSON there, remove the comment. If you want, I can help you turn your whole Helm values file into a clean, valid version using: - one shared IRSA service account, or - split worker/job service accounts.
m
what if I want to assign permissions to specific security groups to my worker
@Marvin what if I want to assign permissions to specific security groups to my worker
m
✅ thought for 20.3 seconds
If you're on EKS with IRSA, security groups are not assigned to service accounts. They are assigned at the pod/network interface level. So there are two different kinds of AWS access controls here: - IAM role via IRSA → controls what AWS APIs the pod can call - Security group → controls network traffic to/from the pod Those are separate. For your Prefect worker specifically: - The worker service account can be annotated with an IAM role - If you want the worker pod to have a specific security group, that is usually done with Security Groups for Pods in EKS, not via the service account annotation alone Important distinction You asked about assigning permissions to specific security groups “to my worker”. That could mean one of two things: 1. You want the worker pod to use a specific AWS IAM role → use IRSA on the Kubernetes service account 2. You want the worker pod to have specific network rules / SGs → use EKS Security Groups for Pods Those are configured differently. If you mean IAM permissions Use this:
Copy code
yaml
serviceAccount:
  create: true
  name: "prefect-worker"
  annotations:
    <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: ${prefect_worker_role_arn}
That gives the worker pod AWS API permissions through the IAM role. If you mean network security group attachment You generally do not attach a security group directly in the Prefect 
baseJobTemplate
as a simple field on the pod spec. In EKS, Security Groups for Pods is an AWS VPC CNI feature. Typically you configure it using: - pod labels/selectors - 
SecurityGroupPolicy
resources - VPC CNI support in the cluster A common pattern is: - label the worker pod - create a 
SecurityGroupPolicy
that matches that label - EKS attaches the desired SG to matching pods That means this is more of an EKS networking configuration than a Prefect configuration. For the worker deployment pod Because the worker itself is created by the Helm chart, you’d typically: - add labels to the worker pod template if the chart supports it - create a 
SecurityGroupPolicy
selecting those labels For flow run job pods Those are created by the Kubernetes worker, so that’s where 
baseJobTemplate
is useful. You can add labels/annotations there so that job pods match a 
SecurityGroupPolicy
. For example, conceptually:
Copy code
yaml
worker:
  config:
    baseJobTemplate:
      configuration: |
        {
          "job_configuration": {
            "job_manifest": {
              "spec": {
                "template": {
                  "metadata": {
                    "labels": {
                      "network-role": "prefect-job"
                    }
                  },
                  "spec": {
                    "serviceAccountName": "prefect-worker"
                  }
                }
              }
            }
          }
        }
Then in Kubernetes/EKS, a 
SecurityGroupPolicy
could target:
Copy code
yaml
podSelector:
  matchLabels:
    network-role: prefect-job
and assign the desired security groups. So can you assign a security group “to the worker”? Yes, but not by service account alone. You’d usually do it through EKS pod networking features, not through Prefect directly. How this maps to Prefect pieces - Worker deployment pod - configured by Helm chart values - security group assignment is cluster/EKS-side - IAM role via service account annotation - Flow run pods - configured via 
baseJobTemplate
- you can add labels/annotations/service account there - security group assignment still happens via EKS mechanisms Best practice - Use IRSA for AWS API permissions - Use Security Groups for Pods for network-level restrictions - Use separate service accounts/roles only if worker and jobs need different AWS API permissions - Use 
baseJobTemplate
mainly to shape the spawned job pods One caution If your goal is “allow the worker to reach RDS/Redis/internal services,” a security group only matters if:
- your cluster supports Security Groups for Pods, and - those destinations are gated by SG-to-SG rules Otherwise you may need Kubernetes 
NetworkPolicy
, node security groups, or both. If you want, I can help with either of these next: 1. a Prefect worker Helm values example for labeling worker/job pods for 
SecurityGroupPolicy
2. an EKS SecurityGroupPolicy manifest example for Prefect worker/job pods 3. a recommended split between IAM role permissions and security group/network permissions
m
@Marvin if this is my container definition
Copy code
WORKDIR /home/prefect
USER prefect
ENV HOME=/home/prefect
ENV PATH=$HOME/.local/bin:$PATH


RUN mkdir -p "$HOME/.pip"
COPY .circleci/pip.conf "$HOME/.pip/pip.conf.template"
RUN sed "s|\$ARTIFACTORY_USER|$ARTIFACTORY_USER|g; s|\$ARTIFACTORY_APIKEY|$ARTIFACTORY_APIKEY|g" \
    "$HOME/.pip/pip.conf.template" > "$HOME/.pip/pip.conf"

ADD requirements.txt .
RUN pip3 install --user -r requirements.txt --break-system-packages && rm -rf "$HOME/.pip/pip.conf"


COPY --chown=prefect:prefect . .
m
✅ thought for 18.9 seconds
m
@marving
m
This Dockerfile setup affects the image runtime user and filesystem, but not the Kubernetes service account. Those are two different things: - Docker 
USER prefect
 → the Linux user inside the container - Kubernetes 
serviceAccountName
 → the Kubernetes identity for the pod, including IRSA on EKS So even if your container runs as:
Copy code
dockerfile
USER prefect
you can still run the pod with this Kubernetes service account:
Copy code
yaml
spec:
  serviceAccountName: prefect-worker
and IRSA will still work. Key point: IRSA is based on the pod’s Kubernetes service account, not the Unix user inside the container. So this is valid: - container runs as Linux user 
prefect
- pod uses Kubernetes service account 
prefect-worker
- AWS credentials are still provided through IRSA What your Dockerfile implies for Prefect Your image is setting:
Copy code
dockerfile
WORKDIR /home/prefect
USER prefect
ENV HOME=/home/prefect
ENV PATH=$HOME/.local/bin:$PATH
That means: - the process runs as non-root user 
prefect
- Python packages are installed into 
/home/prefect/.local
- the container expects to read/write under 
/home/prefect
That’s generally fine for Prefect jobs. What you need to watch out for If your Kubernetes job/pod spec adds security context settings, they need to be compatible with that 
prefect
user. For example, these may matter: - 
runAsUser
- 
runAsGroup
- 
fsGroup
- read-only filesystem - mounted volumes ownership If you don’t set them explicitly, the image’s 
USER prefect
usually works fine. How this relates to your base job template You can still use:
Copy code
json
{
  "job_configuration": {
    "job_manifest": {
      "spec": {
        "template": {
          "spec": {
            "serviceAccountName": "prefect-worker"
          }
        }
      }
    }
  }
}
That tells Kubernetes which service account the pod uses. It does not conflict with:
Copy code
dockerfile
USER prefect
If you want job pods to use IRSA Then the important part is:
Copy code
yaml
serviceAccountName: prefect-worker
or another annotated service account like 
prefect-job
. Not the Docker 
USER
. If you want security groups for pods too Same story: - Docker 
USER prefect
does not affect SG assignment - SG-for-Pods is based on pod networking config / labels / 
SecurityGroupPolicy
Good mental model Think of it like this: - Docker user = who the process is inside the container - Kubernetes service account = who the pod is in the cluster/cloud - IAM role (IRSA) = what AWS APIs the pod may call - Security group = what network traffic the pod may send/receive All four can be different knobs. Example This is perfectly normal: - Docker image: - 
USER prefect
- Kubernetes pod: - 
serviceAccountName: prefect-worker
- Service account annotation: - 
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::123:role/prefect-worker
- Pod gets AWS credentials from IRSA - Process inside pod runs as Linux user 
prefect
One practical caveat If your code or dependencies try to write somewhere like: - 
/root
- system site-packages - protected directories it may fail because your image runs as non-root. But that’s unrelated to service account / IRSA. Given your Dockerfile, this should be fine as long as: - Prefect can start from 
/home/prefect
- any temp/config/cache writes go to writable locations - your flow code doesn’t require root privileges So the short answer is: Yes, you can still use the same Kubernetes service account for the worker/jobs even though your container runs as 
USER prefect
. If you want, I can show you a complete 
baseJobTemplate
example that combines: - 
serviceAccountName
- image - env vars / secret refs - security context - labels for EKS SecurityGroupPolicy
m
@Marvin if this my container definition
Copy code
from reifyhealth/prefect:latest
WORKDIR /home/prefect
USER prefect
ENV HOME=/home/prefect
ENV PATH=$HOME/.local/bin:$PATH


RUN mkdir -p "$HOME/.pip"
COPY .circleci/pip.conf "$HOME/.pip/pip.conf.template"
RUN sed "s|\$ARTIFACTORY_USER|$ARTIFACTORY_USER|g; s|\$ARTIFACTORY_APIKEY|$ARTIFACTORY_APIKEY|g" \
    "$HOME/.pip/pip.conf.template" > "$HOME/.pip/pip.conf"

ADD requirements.txt .
RUN pip3 install --user -r requirements.txt --break-system-packages && rm -rf "$HOME/.pip/pip.conf"


COPY --chown=prefect:prefect . .
and this is my deployment yaml
Copy code
name: prefect3-template
prefect-version: 3.4.11

ost/data-flow: &data-flow
  version: "{{ $CIRCLE_SHA1 }}"
  entrypoint: flows.py:hello_world_flow
  tags: [ "main", "studyteam"]

deployments:
  - name: "k8_hello_world"
    <<: *data-flow
#    schedule:
#      cron: 15 3,15 * * *
    work_pool:
      name: reify-prefect3-{{ $TENANT }}-{{ $ENVIRONMENT }}-k8-pool
      job_variables:
        task_definition_arn: arn:aws:ecs:us-east-1:{{ $ST_ACCOUNT_ID }}:task-definition/prefect3-k8_hello_world-{{ $TENANT }}-{{ $ENVIRONMENT }}


  - name: "ecs_hello_world"
    <<: *data-flow
#    schedule:
#      cron: 15 3,15 * * *
    work_pool:
      name: reify-prefect3-{{ $TENANT }}-{{ $ENVIRONMENT }}-pool
      job_variables:
        task_definition_arn: arn:aws:ecs:us-east-1:{{ $ST_ACCOUNT_ID }}:task-definition/prefect3-ecs_hello_world-{{ $TENANT }}-{{ $ENVIRONMENT }}


pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /home/prefect
Where ecs_hello_world is linking to an ecs pool and k8_hello_world is linking to a kubernetes pool, why does the ECS deployment run fine but I get a
Copy code
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect.deployments.steps.set_working_directory
error in the kubernetes deployment
m
✅ thought for 160.8 seconds
Most likely: your Kubernetes flow run pod does not actually have the code at 
/home/prefect
available the way your ECS run does, so the 
pull
step
Copy code
yaml
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /home/prefect
fails when Prefect tries to 
chdir
into that directory during the run. Important detail: 
set_working_directory
runs inside the runtime container for the flow run, not on your local machine and not in the worker process. So the question becomes: Why does the ECS runtime container have a usable 
/home/prefect
, but the Kubernetes runtime container doesn’t? There are a few likely reasons. --- First thing I notice: your deployment YAML for the Kubernetes pool looks suspicious You said:
Copy code
yaml
- name: "k8_hello_world"
  <<: *data-flow
  work_pool:
    name: reify-prefect3-{{ $TENANT }}-{{ $ENVIRONMENT }}-k8-pool
    job_variables:
      task_definition_arn: arn:aws:ecs:us-east-1:{{ $ST_ACCOUNT_ID }}:task-definition/prefect3-k8_hello_world-{{ $TENANT }}-{{ $ENVIRONMENT }}
task_definition_arn
is an ECS job variable, not a Kubernetes one. So if this deployment points to a Kubernetes work pool, but you are still passing ECS-style 
job_variables
, then your Kubernetes pool is probably not receiving the right runtime configuration at all. That’s a big red flag. For a Kubernetes work pool, the relevant job variables are typically things like: - 
image
- 
namespace
- 
service_account_name
- env / labels / resources / image pull policy, etc. Not 
task_definition_arn
. So the ECS deployment may work because it gets the correct ECS task definition, while the Kubernetes deployment may be falling back to defaults or using an incomplete job spec. --- What 
set_working_directory
actually does This step is basically just:
Copy code
python
os.chdir("/home/prefect")
If that path does not exist in the flow run container, you’ll get a 
StepExecutionError
wrapping something like: - 
FileNotFoundError
- 
PermissionError
The real underlying error is usually in the logs just below that message. If you can, check the full pod logs for the actual cause under the wrapper error. --- Why ECS can work while Kubernetes fails Here are the most likely explanations, in order. 1. Your Kubernetes job is not using the image you think it is Your Dockerfile creates 
/home/prefect
as the working directory and copies code there:
Copy code
dockerfile
WORKDIR /home/prefect
USER prefect
ENV HOME=/home/prefect
COPY --chown=prefect:prefect . .
So if the Kubernetes job were truly running this exact image, 
/home/prefect
should likely exist. If it doesn’t, then the most likely cause is: - ECS deployment is using your custom image - Kubernetes deployment is using a default Prefect image or another image - therefore 
/home/prefect
and/or your code is not there This fits your symptoms very well. For a Kubernetes work pool, make sure the deployment or pool sets the image explicitly. Something like:
Copy code
yaml
work_pool:
  name: reify-prefect3-{{ $TENANT }}-{{ $ENVIRONMENT }}-k8-pool
  job_variables:
    image: your-registry/your-image:{{ $CIRCLE_SHA1 }}
The exact field depends on your pool/job template, but it should be an 
image
-type variable, not 
task_definition_arn
. --- 2. The Kubernetes work pool base job template may override the image or directory behavior If your Kubernetes work pool has a custom 
baseJobTemplate
, that template may define: - a different 
image
- a different command - a different working directory - a pod spec that doesn’t align with your built image If the job template doesn’t use your application image, then 
set_working_directory: /home/prefect
will fail. --- 3. 
/home/prefect
exists in the image, but your code is not there Even if 
/home/prefect
exists, Prefect also needs the deployment entrypoint file to exist relative to that directory:
Copy code
yaml
entrypoint: flows.py:hello_world_flow
That means after 
set_working_directory
succeeds, Prefect expects to find:
Copy code
text
/home/prefect/flows.py
If the code wasn’t copied into the runtime image used by Kubernetes, you may next see import/entrypoint errors. So even if the step error is currently on 
set_working_directory
, the root issue may still be “wrong image / wrong code location”. --- 4. Permissions issue Your image runs as:
Copy code
dockerfile
USER prefect
That is usually fine. But if Kubernetes injects a security context or volume mount that changes permissions, then 
/home/prefect
might exist but be inaccessible. That would usually produce 
PermissionError
, not “No such file or directory”. So this is possible, but less likely than “wrong image”. --- 5. HTML escaping in the pasted config I also see escaped YAML in your pasted example:
Copy code
yaml
<<: *data-flow
and HTML entities like 
>
and 
&
. I’m assuming that’s just Slack formatting and not your actual file. If those are literally in the file, that would break things in strange ways. Probably not the main issue, but worth noting. --- What I think is happening The strongest signal is this:
Copy code
yaml
job_variables:
  task_definition_arn: ...
inside a deployment targeting a Kubernetes work pool. That suggests your ECS deployment is correctly configured, but your Kubernetes deployment is not actually telling the pool to run your custom image. As a result, the K8s pod likely starts from a default image that doesn’t contain your project at 
/home/prefect
. So: - ECS run uses your ECS task definition → custom image with code → works - Kubernetes run uses Kubernetes pool defaults or wrong job config → no code at 
/home/prefect
→ 
set_working_directory
fails --- What to change For the Kubernetes deployment, use Kubernetes-appropriate job variables, especially the image. Conceptually:
Copy code
yaml
- name: "k8_hello_world"
  <<: *data-flow
  work_pool:
    name: reify-prefect3-{{ $TENANT }}-{{ $ENVIRONMENT }}-k8-pool
    job_variables:
      image: your-registry/your-image:{{ $CIRCLE_SHA1 }}
      service_account_name: prefect-worker
If your Kubernetes pool’s base job template defines additional variables, use those exact names. --- How to confirm quickly Check the failing Kubernetes job pod and inspect: - actual image used - whether 
/home/prefect
exists - whether 
flows.py
exists there A quick way is to inspect the job/pod manifest in Kubernetes and compare the container image to the one used in ECS. You want to answer: 1. What image did the K8s job actually run? 2. Does that image contain 
/home/prefect/flows.py
? If the answer to #2 is no, that’s your issue. --- What a valid model looks like For a custom-image deployment to Kubernetes: - your image contains the code - the job uses that image - 
pull
step sets working dir to where the code lives - 
entrypoint: flows.py:hello_world_flow
resolves from there Example:
Copy code
yaml
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /home/prefect

deployments:
  - name: k8_hello_world
    entrypoint: flows.py:hello_world_flow
    work_pool:
      name: my-k8s-pool
      job_variables:
        image: myrepo/myflow:{{ $CIRCLE_SHA1 }}
        service_account_name: prefect-worker
--- Next best debugging step Please paste either: 1. the full underlying error under 
StepExecutionError
, or 2. your Kubernetes work pool 
baseJobTemplate
The exact base job template will tell us what job variables the pool expects, and I can tell you exactly how 
job_variables
should be set for your K8s deployment. Related docs: - Work pools - Kubernetes deployments - Custom images
m
@Marvin I changed to image and that worked. However, the job never went the phase of submittedToInfrastructure. Where can I look for logs for what the problem is?
m
✅ thought for 140.0 seconds
Yes — the first place to look is the worker logs, then the Kubernetes Job/Pod events. If a run never progresses past “submitted to infrastructure” / “submitting”, it usually means one of these happened: - the worker failed while creating the Kubernetes Job - the Job was created but the Pod never started - the Pod started but Prefect didn’t observe the state transition yet Where to check 1. Prefect worker pod logs This is the best first stop.
Copy code
bash
kubectl get pods -n prefect3
kubectl logs -n prefect3 <prefect-worker-pod-name> --tail=200 -f
Look for messages about: - creating Kubernetes jobs - failing to submit infrastructure - RBAC/permission errors - namespace errors - image pull issues - bad job manifest / invalid template If your worker is in another namespace, use that namespace instead. 2. Kubernetes Jobs created by the worker See whether the Job got created at all:
Copy code
bash
kubectl get jobs -A
If you know the namespace where flow runs should launch:
Copy code
bash
kubectl get jobs -n <job-namespace>
If the Job exists, describe it:
Copy code
bash
kubectl describe job <job-name> -n <job-namespace>
3. Kubernetes Pods for that Job If the Job exists, get the Pod:
Copy code
bash
kubectl get pods -n <job-namespace>
kubectl describe pod <pod-name> -n <job-namespace>
kubectl logs <pod-name> -n <job-namespace>
If the pod never starts, 
describe pod
is usually the most useful command. It will show events like: - 
ImagePullBackOff
- 
ErrImagePull
- 
CreateContainerConfigError
- 
FailedScheduling
- 
Forbidden
- missing service account - missing secret/configmap Common causes when stuck after submission For Kubernetes work pools, the usual culprits are: - wrong namespace - worker RBAC can’t create Jobs/Pods - job service account doesn’t exist - referenced secret doesn’t exist - private image can’t be pulled - bad base job template - resource requests too high / pod unschedulable - invalid image name Given your recent change to 
image
, I’d especially check for: - bad image tag - missing imagePullSecret - image not accessible from cluster Very useful commands
Copy code
bash
kubectl get events -n <job-namespace> --sort-by='.lastTimestamp'
and
Copy code
bash
kubectl describe pod <pod-name> -n <job-namespace>
Those often tell you the exact reason immediately. Also check in Prefect UI / CLI You can inspect the flow run and stream logs:
Copy code
bash
prefect flow-run inspect <flow-run-id>
and if available in your environment:
Copy code
bash
prefect flow-run logs <flow-run-id>
The CLI help for these timed out on my side, so if 
flow-run logs
isn’t available in your installed version, use the UI or worker/pod logs directly. Practical debugging order I’d use 1. 
kubectl logs
for the worker 2. 
kubectl get jobs -n <ns>
3. 
kubectl get pods -n <ns>
4. 
kubectl describe pod ...
5. 
kubectl get events -n <ns> --sort-by='.lastTimestamp'
What I’d expect in your case Since changing to 
image
fixed the earlier issue, the next failure is often one of: - the Kubernetes job is using a private image without pull credentials - the service account in the job template is missing - the namespace/service account/secret referenced by the template doesn’t exist in the job namespace If you want, paste: - your current Kubernetes 
baseJobTemplate
- the output of 
kubectl describe pod <pod-name>
- or the worker log snippet and I can help pinpoint the exact failure.
33 Views
Open in Slack
PreviousNext
Powered by