Short answer: - In Prefect 3.x with a Kubernetes work pool/worker, you get one Kubernetes Job/Pod per flow run (not per task). The dashboard’s “tasks” are not 1:1 with pods. - Prefect does not delete Jobs/Pods. Kubernetes will only clean them up if you set a TTL on the Job. If TTL is not set, completed/failed/cancelled Jobs/Pods will accumulate and you’ll see a large count in the cluster even when the Prefect UI shows few active runs. Why you’re seeing a mismatch - Pod-per-flow-run model: All tasks in a flow run execute inside a single pod. So pod counts are unrelated to the number of task runs you see in the UI. - Jobs retained by default: If your work pool/template does not set a TTL (ttlSecondsAfterFinished), completed/failed/cancelled Jobs (and their pods) remain until you delete them. This is the most common cause of “hundreds of pods/jobs” but only a handful of active runs in the Prefect UI. - Re-schedules can multiply Jobs: If a pod is evicted or fails and the flow run is rescheduled, a new Job is created for the same flow run. You can end up with multiple Jobs for the same Prefect flow run. Look for the label

<http://prefect.io/flow-run-id|prefect.io/flow-run-id>

on Jobs to see this. - Kubernetes retries: If your Job template sets

backoffLimit > 0

, K8s may create additional pods for retries within the same Job, inflating pod counts. What to do 1) Enable automatic cleanup with a TTL - In your Kubernetes work pool, set “Finished Job TTL” (e.g., 3600 seconds). This injects

spec.ttlSecondsAfterFinished

into the Job so Kubernetes deletes it (and its pods) after the TTL. - If you use a custom Job template, make sure it includes:

spec:
  ttlSecondsAfterFinished: "{{ finished_job_ttl }}"

Then set

finished_job_ttl

on the work pool to a reasonable value (e.g., 3600). 2) Consider Kubernetes retry behavior - If you want Prefect to handle retries/reschedules, set

backoffLimit: 0

in your Job template. That avoids extra pods created by K8s retries. 3) Verify what’s accumulating - See statuses of Jobs/Pods:

kubectl get jobs,pods -n <your-namespace>

- Check how many Jobs exist per Prefect flow run:

kubectl get jobs -n <your-namespace> -l <http://prefect.io/flow-run-id|prefect.io/flow-run-id> -o custom-columns=NAME:.metadata.name,FLOW_RUN_ID:.metadata.labels.prefect\.io/flow-run-id,SUCCEEDED:.status.succeeded,FAILED:.status.failed,ACTIVE:.status.active

If you see many completed Jobs for the same

flow-run-id

, that’s reschedules or retained history. Setting TTL will clean these up going forward. Helpful docs - Kubernetes integration overview: prefect-kubernetes docs - Work pools overview: Work pools in Prefect 3 A couple of clarifying questions to make sure we zero in on your setup: - Which Prefect version are you on (assuming 3.x)? - Do you have “Finished Job TTL” set in your Kubernetes work pool? - What’s your Job template

backoffLimit

? - Are the 500 pods/jobs mostly in Completed/Failed status, or many are still Running?

Got it — this lines up with your config: - Prefect 3 - Kubernetes work pool/worker - backoffLimit=0 - No TTL configured - ~500 Jobs/Pods, mostly Completed, some weeks old What’s happening - By default, Prefect does not delete Kubernetes Jobs/Pods. If TTL is not set, finished Jobs (Succeeded/Failed/Cancelled) stick around indefinitely. That’s why weeks-old Jobs are still in your cluster even though the UI only shows a few active runs. - With backoffLimit=0, you won’t get K8s retries (good for Prefect-managed retries), but it doesn’t affect cleanup. How to fix going forward (set a TTL) - UI: Work Pools -> your Kubernetes pool -> Variables -> set “Finished Job TTL” (e.g., 3600 seconds). This adds

spec.ttlSecondsAfterFinished

to new Jobs so Kubernetes cleans them up after an hour. - If you use a custom job template, make sure it includes:

spec:
  ttlSecondsAfterFinished: "{{ finished_job_ttl }}"

Then set

finished_job_ttl

on the work pool. CLI option (if you prefer editing the base job template)

# Fetch default K8s template
prefect work-pool get-default-base-job-template --type kubernetes > k8s-job-template.json

# Edit k8s-job-template.json to include:
# "spec": { "ttlSecondsAfterFinished": "{{ finished_job_ttl }}", ... }

# Apply to an existing pool
prefect work-pool update "YOUR_POOL_NAME" --base-job-template k8s-job-template.json

Note: TTL only applies to newly created Jobs. Old Jobs won’t be cleaned up retroactively. One-time cleanup of old Jobs - If the namespace is dedicated to Prefect (or you’re sure no runs are currently active), a simple, broad cleanup:

kubectl delete job -n <namespace> -l <http://prefect.io/flow-run-id|prefect.io/flow-run-id>

- If you want to only delete finished Jobs and keep any active ones, use a safer filter (requires jq):

kubectl get jobs -n <namespace> -l <http://prefect.io/flow-run-id|prefect.io/flow-run-id> -o json \
| jq -r '.items[] | select((.status.active|tonumber) != 1 and ((.status.succeeded|tonumber) >= 1 or (.status.failed|tonumber) >= 1)) | .metadata.name' \
| xargs -r -n1 kubectl delete job -n <namespace>

Deleting a Job will also delete its controlled pods. Recommendations - Keep backoffLimit at 0 (let Prefect handle retries/reschedules). - Set Finished Job TTL (e.g., 3600–86400) to prevent future accumulation. Docs - Kubernetes integration: prefect-kubernetes docs - Work pools in Prefect 3: Work pools overview If you share your pool name/namespace, I can tailor a cleanup command that won’t touch any currently running Jobs.