https://prefect.io logo
Join Slack
Powered by
how to terminate automatically succeeded and faile...
# marvin-ai
v
how to terminate automatically succeeded and failed jobs after 60 mins in prefect @Marvin
m
✅ thought for 463.4 seconds
Great question. The answer depends on which infrastructure your Prefect work pool uses. If you’re on Kubernetes, you can use the cluster’s native TTL to automatically delete finished Jobs; if you’re on Docker or Process, the behavior differs. If you’re using Kubernetes (most likely given “jobs”): - Set the Kubernetes Job TTL to 3600 seconds (60 minutes). In Prefect’s Kubernetes work pools, this is the 
finished_job_ttl
variable which maps to the Job spec’s 
spec.ttlSecondsAfterFinished
. - You can set it at the work pool level (applies to all deployments in the pool) or override it per deployment. Ways to set it: 1) Prefect UI - Work Pools → your Kubernetes pool → Edit - Set “Finished Job TTL” to 
3600
- Save - Docs: Kubernetes work pool guide 2) Per-deployment in code (Prefect 3.x)
Copy code
from prefect import flow

@flow
def my_flow():
    ...

# If deploying from source:
my_flow.from_source(
    "<https://github.com/owner/repo>",
    "path/to/file.py:my_flow",
).deploy(
    name="prod",
    work_pool_name="my-k8s-pool",
    job_variables={"finished_job_ttl": 3600},  # 60 minutes
)
3) CLI via a base job template - Get the default template, set 
finished_job_ttl
to 3600, then update the pool:
Copy code
prefect work-pool get-default-base-job-template --type kubernetes > base-job-template.json
# Edit base-job-template.json and set the variables.finished_job_ttl default to 3600
prefect work-pool update "my-k8s-pool" --base-job-template base-job-template.json
- CLI reference: work-pool CLI Notes: - This deletes both Completed and Failed Jobs after the TTL, handled by Kubernetes’ TTL Controller. Ensure it’s enabled in your cluster: Kubernetes TTL Controller docs. - This controls cleanup of Kubernetes Job resources; it doesn’t affect Prefect’s flow/task run records in the UI. If you’re using Docker work pools: - There’s no “delay then delete” TTL. You can set 
auto_remove: true
to remove containers immediately after they exit: - Per-deployment: 
job_variables={"auto_remove": True}
- Work pool default: put 
auto_remove
in the pool’s default job variables - If you truly need “delete 60 minutes after exit” for Docker, you’d need an external cleanup (e.g., a cron job with 
docker system prune
policy). Prefect does not provide a delayed cleanup for Docker containers. If this isn’t Kubernetes, let me know which work pool type you’re using (Kubernetes, Docker, ECS, Process, etc.) and I’ll tailor the exact steps.
v
@Marvin how to set this Ways to set it: 1) Prefect UI - Work Pools → your Kubernetes pool → Edit - Set “Finished Job TTL” to 
3600
- Save - Docs: Kubernetes work pool guide
m
✅ thought for 297.6 seconds
Great question — in Prefect 3.x this setting is controlled by the 
finished_job_ttl
variable on your Kubernetes work pool’s base job template. It renders to the Kubernetes Job field 
spec.ttlSecondsAfterFinished
. In the UI it’s labeled “Finished Job TTL”. Ways to set it to 3600 seconds: 1) Prefect UI - Work Pools → your Kubernetes pool → Edit - Set “Finished Job TTL” to 
3600
- Save - Docs: Kubernetes work pool guide 2) CLI - Export the default Kubernetes base job template, set the default, then update the pool:
Copy code
prefect work-pool get-default-base-job-template --type kubernetes --file base-job-template.json

# Edit base-job-template.json:
# variables.properties.finished_job_ttl.default = 3600
# (If the variable is missing, add it, and ensure job_configuration.spec.ttlSecondsAfterFinished = "{{ finished_job_ttl }}")

prefect work-pool update "your-k8s-pool" --base-job-template base-job-template.json
- Creating a new pool with it set:
Copy code
prefect work-pool create "your-k8s-pool" --type kubernetes --base-job-template base-job-template.json
3) Python
Copy code
import asyncio
from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import WorkPoolUpdate

async def set_finished_job_ttl(pool_name: str, seconds: int = 3600):
    async with get_client() as client:
        pool = await client.read_work_pool(pool_name)
        tpl = pool.base_job_template

        # Ensure the variable exists, then set the default
        tpl.setdefault("variables", {}).setdefault("properties", {})
        tpl["variables"]["properties"].setdefault("finished_job_ttl", {
            "title": "Finished Job TTL",
            "type": "integer",
            "description": "Seconds to retain Kubernetes Jobs after completion."
        })
        tpl["variables"]["properties"]["finished_job_ttl"]["default"] = seconds

        # Ensure it's wired into the Job spec (if not already)
        tpl.setdefault("job_configuration", {}).setdefault("spec", {}) \
           .update({"ttlSecondsAfterFinished": "{{ finished_job_ttl }}"})

        await client.update_work_pool(
            work_pool_name=pool_name,
            work_pool=WorkPoolUpdate(base_job_template=tpl),
        )

asyncio.run(set_finished_job_ttl("your-k8s-pool", 3600))
Naming recap: - UI label: “Finished Job TTL” - Base job template variable: 
finished_job_ttl
(seconds) - Kubernetes Job field: 
spec.ttlSecondsAfterFinished
Note: Kubernetes will garbage-collect Jobs after this TTL. Omitting the field keeps Jobs indefinitely.
v
give me this vailbel how to configure finished_job_ttl @Marvin
m
✅ thought for 32.6 seconds
Here’s exactly how to configure the 
finished_job_ttl
variable for a Kubernetes work pool (Prefect 3.x). This controls 
spec.ttlSecondsAfterFinished
on the Kubernetes Job. What it is - Variable name: 
finished_job_ttl
(seconds) - UI label: “Finished Job TTL” - Rendered in Job spec as: 
spec.ttlSecondsAfterFinished
Set it via the UI - Work Pools → your Kubernetes pool → Edit → set “Finished Job TTL” to the number of seconds (e.g., 
3600
) → Save - Docs: Kubernetes work pool guide Set it via the CLI - Export the default base job template, set the variable, update the pool:
Copy code
prefect work-pool get-default-base-job-template --type kubernetes --file base-job-template.json

# Edit base-job-template.json:
# 1) Ensure this variable exists and set a default:
# "variables": {
#   "type": "object",
#   "properties": {
#     "finished_job_ttl": {
#       "title": "Finished Job TTL",
#       "description": "Seconds to retain Kubernetes Jobs after completion.",
#       "type": "integer",
#       "default": 3600
#     }
#   }
# }

# 2) Ensure the Job spec wires it in:
# "job_configuration": {
#   "spec": {
#     "ttlSecondsAfterFinished": "{{ finished_job_ttl }}"
#   }
# }

prefect work-pool update "your-k8s-pool" --base-job-template base-job-template.json
- Creating a new pool with it set:
Copy code
prefect work-pool create "your-k8s-pool" --type kubernetes --base-job-template base-job-template.json
Minimal JSON you can paste into your template - Variable definition:
Copy code
"variables": {
  "type": "object",
  "properties": {
    "finished_job_ttl": {
      "title": "Finished Job TTL",
      "description": "Seconds to retain Kubernetes Jobs after completion.",
      "type": "integer",
      "default": 3600
    }
  }
}
- Job spec wiring:
Copy code
"job_configuration": {
  "spec": {
    "ttlSecondsAfterFinished": "{{ finished_job_ttl }}"
  }
}
Set it via Python
Copy code
import asyncio
from prefect.client.orchestration import get_client
from prefect.client.schemas.actions import WorkPoolUpdate

async def set_finished_job_ttl(pool_name: str, seconds: int = 3600):
    async with get_client() as client:
        pool = await client.read_work_pool(pool_name)
        tpl = pool.base_job_template

        tpl.setdefault("variables", {}).setdefault("properties", {})
        tpl["variables"]["properties"].setdefault("finished_job_ttl", {
            "title": "Finished Job TTL",
            "type": "integer",
            "description": "Seconds to retain Kubernetes Jobs after completion."
        })
        tpl["variables"]["properties"]["finished_job_ttl"]["default"] = seconds

        tpl.setdefault("job_configuration", {}).setdefault("spec", {}) \
           .update({"ttlSecondsAfterFinished": "{{ finished_job_ttl }}"})

        await client.update_work_pool(
            work_pool_name=pool_name,
            work_pool=WorkPoolUpdate(base_job_template=tpl),
        )

asyncio.run(set_finished_job_ttl("your-k8s-pool", 3600))
Notes - Value is in seconds. If omitted, Jobs are retained indefinitely by Kubernetes.
v
@Marvin please check this do i have right config " "finished_job_ttl": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "title": "TTL Seconds After Finished", "default": 7200 }, "image_pull_policy": { "enum": [ "IfNotPresent", "Always", "Never" ], "type": "string", "title": "Image Pull Policy", "default": "IfNotPresent" }, "image_pull_secret": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Image Pull Secret", "default": "acr-secret" }, "service_account_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Service Account Name", "default": "prefect-worker" }, "job_watch_timeout_seconds": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "title": "Job Watch Timeout Seconds" }, "pod_watch_timeout_seconds": { "type": "integer", "title": "Pod Watch Timeout Seconds" } } }, "job_configuration": { "env": "{{ env }}", "name": "{{ name }}", "labels": "{{ labels }}", "command": "{{ command }}", "namespace": "{{ namespace }}", "job_manifest": { "kind": "Job", "spec": { "template": { "spec": { "containers": [ { "env": "{{ env }}", "args": "{{ command }}", "name": "prefect-job", "image": "{{ image }}", "imagePullPolicy": "{{ image_pull_policy }}" } ], "restartPolicy": "Never", "imagePullSecrets": [ { "name": "{{ image_pull_secret }}" } ], "serviceAccountName": "{{ service_account_name }}" } }, "completions": 1, "parallelism": 1, "backoffLimit": "{{ backoff_limit }}", "ttlSecondsAfterFinished": "{{ finished_job_ttl }}" }, "metadata": { "labels": "{{ labels }}", "namespace": "{{ namespace }}", "generateName": "{{ name }}-" }, "apiVersion": "batch/v1" }, "stream_output": "{{ stream_output }}", "job_watch_timeout_seconds": "{{ job_watch_timeout_seconds }}", "pod_watch_timeout_seconds": "{{ pod_watch_timeout_seconds }}" } }"
m
✅ thought for 76.8 seconds
Almost there! Your 
finished_job_ttl
variable is defined correctly, and you’re wiring it to the Job field 
spec.ttlSecondsAfterFinished
. Two important fixes: - Don’t quote numeric placeholders. Quote-wrapped values become strings (e.g., "7200") and Kubernetes expects integers. - Do the same for other numeric fields like 
backoffLimit
, 
job_watch_timeout_seconds
, and 
pod_watch_timeout_seconds
. What to change - From:
Copy code
"ttlSecondsAfterFinished": "{{ finished_job_ttl }}"
"backoffLimit": "{{ backoff_limit }}"
"job_watch_timeout_seconds": "{{ job_watch_timeout_seconds }}"
"pod_watch_timeout_seconds": "{{ pod_watch_timeout_seconds }}"
- To:
Copy code
"ttlSecondsAfterFinished": {{ finished_job_ttl }}
"backoffLimit": {{ backoff_limit }}
"job_watch_timeout_seconds": {{ job_watch_timeout_seconds }}
"pod_watch_timeout_seconds": {{ pod_watch_timeout_seconds }}
Your variable block is fine (you can keep your custom title, or rename it to “Finished Job TTL” if you prefer):
Copy code
"finished_job_ttl": {
  "anyOf": [
    { "type": "integer" },
    { "type": "null" }
  ],
  "title": "TTL Seconds After Finished",
  "default": 7200
}
Small optional note - If 
image_pull_secret
can be null, the manifest will render 
"name": null
, which some clusters reject. Either keep a valid default (like your 
"acr-secret"
) or add conditional templating to omit 
imagePullSecrets
when unset. Reference: Kubernetes work pool guide
v
give me the working template for spec.ttlSecondsAfterFinished @Marvin
m
✅ thought for 116.2 seconds
Here’s a minimal, working base job template snippet that correctly wires 
finished_job_ttl
to 
spec.ttlSecondsAfterFinished
for a Kubernetes work pool in Prefect 3.x. You can paste this into your pool’s base job template (or merge into your existing one):
Copy code
{
  "job_configuration": {
    "apiVersion": "batch/v1",
    "kind": "Job",
    "metadata": {
      "labels": "{{ labels }}",
      "namespace": "{{ namespace }}",
      "generateName": "{{ name }}-"
    },
    "spec": {
      "backoffLimit": "{{ backoff_limit }}",
      "ttlSecondsAfterFinished": "{{ finished_job_ttl }}",
      "template": {
        "spec": {
          "restartPolicy": "Never",
          "serviceAccountName": "{{ service_account_name }}",
          "containers": [
            {
              "name": "prefect-job",
              "env": "{{ env }}",
              "image": "{{ image }}",
              "imagePullPolicy": "{{ image_pull_policy }}",
              "args": "{{ command }}"
            }
          ]
        }
      }
    }
  },
  "variables": {
    "type": "object",
    "properties": {
      "namespace": {
        "type": "string",
        "default": "default"
      },
      "image": {
        "type": "string"
      },
      "image_pull_policy": {
        "type": "string",
        "enum": [
          "IfNotPresent",
          "Always",
          "Never"
        ],
        "default": "IfNotPresent"
      },
      "service_account_name": {
        "type": "string",
        "default": "prefect-worker"
      },
      "backoff_limit": {
        "type": "integer",
        "default": 0
      },
      "finished_job_ttl": {
        "anyOf": [
          {
            "type": "integer"
          },
          {
            "type": "null"
          }
        ],
        "title": "Finished Job TTL",
        "description": "Seconds before Kubernetes cleans up finished Jobs. Use null to retain indefinitely.",
        "default": 3600
      }
    }
  }
}
How this works - 
ttlSecondsAfterFinished
is set to 
"{{ finished_job_ttl }}"
in the manifest. Prefect’s templating preserves the integer type when the value is a standalone placeholder, so Kubernetes receives an integer (e.g., 3600), not a string. - If you set 
finished_job_ttl
to 
null
, the field will render as 
null
, which means Kubernetes will not automatically clean up finished Jobs (they’ll be retained). Docs for reference: Kubernetes work pool guide
74 Views
Open in Slack
PreviousNext
Powered by