Short answer: your worker’s service account is in namespace “um-demo-v3” but the work pool is launching Jobs in the “default” namespace. Either change the work pool’s job namespace to “um-demo-v3” (recommended) or give that service account RBAC to create Jobs in “default”. What to change 1) Set the job namespace (preferred) - In your Kubernetes work pool, set the Variables → namespace = um-demo-v3. This makes the worker create Jobs in that namespace instead of default. - You can also set it per-deployment:

# prefect.yaml
deployments:
  - name: my-flow
    entrypoint: flows/my_flow.py:flow
    work_pool:
      name: <your-k8s-pool>
      job_variables:
        namespace: "um-demo-v3"
        service_account_name: "prefect-k3s-service"

- If you need to inspect or customize the base job template:

prefect work-pool get-default-base-job-template --type kubernetes > k8s.json
# edit if needed, then:
prefect work-pool update "<your-k8s-pool>" --base-job-template k8s.json

2) Or, grant RBAC in the target namespace If you truly want Jobs to run in default, bind permissions for your service account there. Role in default with RoleBinding to SA in um-demo-v3:

apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-worker-role
  namespace: default
rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create","delete","get","list","patch","watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
# If your worker creates a secret for the API key, include:
# - apiGroups: [""]
#   resources: ["secrets"]
#   verbs: ["create","get","patch","delete"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-worker-rolebinding
  namespace: default
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-worker-role
subjects:
- kind: ServiceAccount
  name: prefect-k3s-service
  namespace: um-demo-v3

Quick checks - Which namespace do you expect Jobs to run in? If it’s um-demo-v3, set the work pool namespace as above. - Verify RBAC:

kubectl auth can-i create jobs \
  --as=system:serviceaccount:um-demo-v3:prefect-k3s-service \
  --namespace um-demo-v3

kubectl auth can-i create jobs \
  --as=system:serviceaccount:um-demo-v3:prefect-k3s-service \
  --namespace default

Notes - By default, the Kubernetes work pool uses namespace “default” if you don’t set it. - There’s nothing in the Helm values that changes the job namespace; it’s controlled by the work pool/deployment job variables. In Helm, ensure your worker Pod uses the intended SA and RBAC (or set rbac.create + serviceAccount.name). - Prefect Helm worker chart: prefect-worker Helm chart - Kubernetes worker docs: Prefect Kubernetes integration docs - Work pools concept: Work pools docs If you share which namespace you want to target, I can tailor the exact values, RBAC, and work pool/deployment config.

Nice progress — that message means the worker successfully created the Kubernetes Job. From here, the problem is almost always inside the Job/Pod (image, credentials, code import, or scheduling), not the worker. Fastest way to see what happened 1) Find the Job and Pod in the namespace where you’re launching (sounds like um-demo-v3 now):

kubectl -n um-demo-v3 get jobs
kubectl -n um-demo-v3 describe job <job-name>

kubectl -n um-demo-v3 get pods -l job-name=<job-name>
kubectl -n um-demo-v3 describe pod <pod-name>
kubectl -n um-demo-v3 logs <pod-name> -c prefect-job --tail=200

- If the pod never starts, check Events in describe output (image pull error, scheduling, permission). - If the container starts then exits immediately, logs will usually show the exact Python error (e.g., import/module not found, missing PREFECT_API_KEY, etc.). Common root causes and fixes - Missing Prefect API credentials in the job - Symptom: pod logs show authentication/authorization error or exit before any flow logs. - Fix options: - Easiest/safest: set the worker to create a Secret and wire it automatically: - Set env on the worker:

PREFECT_INTEGRATIONS_KUBERNETES_WORKER_CREATE_SECRET_FOR_API_KEY=true

- Ensure the worker SA has permissions to create/delete Secrets in the job namespace. - Or inject API URL/key via the work pool’s job variables (edit base job template to add env for

PREFECT_API_URL

and

PREFECT_API_KEY

, or reference an existing Secret). - Image pull issues - Symptom:

ImagePullBackOff

or

ErrImagePull

in pod events. - Fix: set

image_pull_secrets

in the work pool/base job template and make sure the secret exists in um-demo-v3. Also verify the image name/tag is correct (default is

prefecthq/prefect:3-latest

if you didn’t override). - Code not present in the image - Symptom:

ModuleNotFoundError

or can’t import your flow module. - Fix: deliver code to the runtime either by - Using

flow.from_source("<https://github.com/owner/repo>")

when deploying, or - Building a custom image that contains your code and dependencies and using that image in the work pool. - Missing Python deps - Symptom: ImportError for third‑party packages. - Fix: include deps in your image or install at runtime (not ideal for K8s jobs). - Scheduling/resource issues - Symptom: pod pending with events like insufficient CPU/memory or node selector/toleration mismatch. - Fix: adjust resource requests/limits or scheduling constraints in the job template. Turn on more logging for diagnosis - Add to your job env via work pool job variables:

env:
  - name: PREFECT_LOGGING_LEVEL
    value: DEBUG

- Tail worker logs to see submission + any observer messages:

kubectl -n um-demo-v3 logs deploy/<your-worker-release> -f

Note: if your worker SA doesn’t have pods/watch and jobs/watch, the worker can’t report pod status events — you’ll still be able to see everything via kubectl, but the worker won’t add extra details after “Completed submission…”. If you share: - The namespace you’re launching into (sounds like um-demo-v3), - The Job name, and the output of: -

kubectl -n um-demo-v3 describe pod <pod>

-

kubectl -n um-demo-v3 logs <pod> -c prefect-job --tail=200

I can pinpoint the exact cause and give you the minimal change to fix it. References - Prefect Kubernetes integration docs: Prefect Kubernetes integration docs - Work pools overview: Work pools docs

✅ thought for 181.8 seconds

Short answer: set the Kubernetes work pool’s default “job namespace” to your namespace. That’s what the worker uses when it creates Jobs. The worker pod’s own namespace (where it runs) is separate. Ways to set it so it’s correct by default: - In the UI (easiest): Work Pools → your Kubernetes pool → Settings/Variables → Namespace = bgriffiths → Save. All deployments using this pool will inherit it. - Via CLI with a base job template: 1) Export the default template:

prefect work-pool get-default-base-job-template --type kubernetes --file k8s-base.json

2) Edit k8s-base.json and change the default:

... "variables": { "namespace": { "default": "bgriffiths", ... } } ...

3) Apply to an existing pool:

prefect work-pool update "<your-pool-name>" --base-job-template k8s-base.json

Or when creating a new pool:

prefect work-pool create "<your-pool-name>" --type kubernetes --base-job-template k8s-base.json

Per-deployment override (optional): - If you only want to change it for one deployment, set

job_variables.namespace

when deploying:

prefect deploy --name my-deployment --pool "<your-pool-name>" --job-variable namespace=bgriffiths

or in prefect.yaml:

deployments:
  - name: my-deployment
    work_pool:
      name: "<your-pool-name>"
      job_variables:
        namespace: "bgriffiths"

Notes: - There is no

prefect worker start

flag or environment variable to set the default job namespace; it must come from the work pool configuration (or deployment-level

job_variables

). - Ensure the worker’s service account has RBAC to create Jobs in the target namespace (bgriffiths). If the worker runs in a different namespace, bind that service account to a Role in

bgriffiths

with permissions for batch/jobs, pods, and pods/log (and secrets if you enable API key secret creation).