Hello <@ULVA73B9P>! How can I automatically delete...
# ask-marvin
a
Hello @Marvin! How can I automatically delete the underlying Kubernetes job when I cancel a job N
m
thought for 370.1 seconds
Great question! Assuming you’re running Prefect 3.x with a Kubernetes work pool, the most reliable way to auto-clean Kubernetes Jobs (and their Pods) when you cancel a flow run is to let Kubernetes handle it via the Job TTL controller. What to do - Add ttlSecondsAfterFinished to the Job spec in your work pool’s base job template. This tells Kubernetes to delete the Job automatically after it reaches a terminal state (Succeeded/Failed). A canceled flow run will cause the Job to finish, then Kubernetes deletes it. - 0 = delete immediately - e.g. 300 = delete 5 minutes after finish Steps 1) Inspect your work pool to get the current base job template:
Copy code
prefect work-pool inspect "my-k8s-pool"
2) Save and edit the base job template JSON to include ttlSecondsAfterFinished under job.spec:
Copy code
{
  "job": {
    "apiVersion": "batch/v1",
    "kind": "Job",
    "metadata": {
      "labels": {
        "<http://prefect.io/work-pool|prefect.io/work-pool>": "my-k8s-pool"
      }
    },
    "spec": {
      "ttlSecondsAfterFinished": 0,
      "backoffLimit": 0,
      "template": {
        "spec": {
          "restartPolicy": "Never",
          "terminationGracePeriodSeconds": 5,
          "containers": [
            {
              "name": "prefect-job",
              "image": "your-image:tag"
            }
          ]
        }
      }
    }
  }
}
3) Update the work pool with your edited template:
Copy code
prefect work-pool create "my-k8s-pool" --type kubernetes --base-job-template ./base-job-template.json --overwrite
Notes and tips - The TTL controller is enabled by default on modern Kubernetes versions. If your cluster is old or managed differently, verify it’s enabled. - When you click “Cancel” in Prefect, the Kubernetes job should terminate; TTL then removes the Job (and the Pods via ownerReferences/GC). Setting a small terminationGracePeriodSeconds helps jobs exit quickly. - There isn’t a Prefect 3.x flag like keep_job to flip this behavior; the recommended pattern is to manage cleanup via the Kubernetes Job spec. - If you prefer some delay for troubleshooting, set ttlSecondsAfterFinished to a few minutes instead of 0. Docs - Deploying to Kubernetes (Prefect 3) If you’re already doing this and still see Jobs sticking around after cancel, let me know: - Your Prefect version - Kubernetes version - The work pool’s base job template snippet - Whether you want immediate deletion or a short grace period for debugging logs