Short answer: you’ll need a “hard kill” outside of Python. In Prefect 2.x,

timeout_seconds

is cooperative and cannot stop a hung C/CUDA call. The reliable patterns are to let Kubernetes or an external watchdog kill the process/pod, and make sure the Prefect Kubernetes worker sees that kill and marks the run failed. First, a quick clarification: Prefect 2.x does not use Kopf anywhere. The Kubernetes worker relies on the Kubernetes API (kubernetes_asyncio) to watch Jobs/Pods; there is no Kopf-based operator. If something “crashes trying to report the crash,” it’s likely the worker’s watch loop or error handling around the Job/Pod events. Links below. Recommended “last-resort” hard-stop approaches (you can mix and match): 1) K8s-level hard wall clock (activeDeadlineSeconds) - Add

spec.activeDeadlineSeconds

to your Job manifest so K8s force-terminates the pod regardless of Python state. - Pair it with a short

terminationGracePeriodSeconds

to reduce how long K8s waits before SIGKILL. - Make Prefect’s worker watch outlast Kubernetes’ deadline so it sees the failure and reports it: - Set worker variable

job_watch_timeout_seconds

a bit longer than

activeDeadlineSeconds + terminationGracePeriodSeconds

(e.g., deadline 3600s + grace 15s → watch ≈ 3660–3720s). - Keep

restartPolicy: Never

and

backoffLimit: 0

. Example Job manifest template in your Kubernetes work pool (Advanced template):

apiVersion: batch/v1
kind: Job
metadata:
  generateName: "{{ name }}-"
  namespace: "{{ namespace }}"
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 3600
  ttlSecondsAfterFinished: "{{ finished_job_ttl }}"
  template:
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 15
      containers:
      - name: prefect-job
        image: "{{ image }}"
        imagePullPolicy: "{{ image_pull_policy }}"
        env: {{ env }}
        # add your resources if needed
        resources:
          limits:
            cpu: "4"
            memory: "16Gi"

2) Subprocess watchdog inside your task (hard-kill the child) - Run the CUDA/C-extension code in a child process and enforce a hard timeout by SIGKILL’ing the process group if it’s still running. This bypasses Python’s cooperative timeout and will kill even if the child is hung in native code. - Disable Prefect’s cooperative timeout on that task and let your watchdog do the killing. Example:

from prefect import task

@task(timeout_seconds=None)  # let the watchdog handle timeout
def gpu_task(cmd: list[str], timeout: int = 3600):
    import os, signal, subprocess
    # Start in its own process group so we can kill all children
    proc = subprocess.Popen(cmd, preexec_fn=os.setsid)
    try:
        proc.wait(timeout=timeout)
    except subprocess.TimeoutExpired:
        # Kill the whole group hard
        os.killpg(proc.pid, signal.SIGKILL)
        raise RuntimeError(f"Timed out after {timeout}s; subprocess SIGKILLed")
    if proc.returncode != 0:
        raise RuntimeError(f"Subprocess exited with {proc.returncode}")

3) Liveness probe as a timer kill (K8s will terminate the container) - If you can’t change your entrypoint to wrap it with

timeout

, you can use a livenessProbe that fails after N seconds, causing K8s to kill the container. - With

restartPolicy: Never

and

backoffLimit: 0

, the Pod will be marked Failed and the Job will fail. Example (kills at ~3600s):

livenessProbe:
  initialDelaySeconds: 3600
  periodSeconds: 5
  failureThreshold: 1
  exec:
    command: ["sh", "-c", "exit 1"]

4) Resource-based hard stops - Set realistic memory/CPU limits so runaway behavior becomes OOMKilled or throttled. OOMKill is a reliable hard-stop signal that Prefect’s worker will treat as failure. Avoiding “ghost requeue” when K8s kills the pod - Ensure your Prefect Kubernetes worker version is up to date (2.x latest) and you’re using the Kubernetes worker (not older “agent” patterns).

- Set

job_watch_timeout_seconds

> any K8s kill threshold so the worker remains alive to see the pod termination and report a final failed state. - Keep

backoffLimit: 0

and

restartPolicy: Never

. - Verify you don’t have flow-level retries or external automations re-creating the run. - If you’re still seeing “ghost” runs, please share: - Exact Prefect version and prefect-kubernetes version - Your work pool variables (esp. job_watch_timeout_seconds, pod_watch_timeout_seconds) - Your Job manifest template - Worker logs around the time the pod is killed Useful references (2.x) - Kubernetes worker source (manifest, watches, job deletion): - Worker implementation: prefect_kubernetes/worker.py - Pod event replication: prefect_kubernetes/events.py - Concepts and guides: - Work pools/workers concepts: Work pools & workers (2.x) - Kubernetes deployment guide: Kubernetes deployment (2.x) - Kubernetes integration docs: prefect-kubernetes (2.x) If you share your current manifest and work pool settings, I’m happy to help tailor the exact values (activeDeadlineSeconds, grace period, and job_watch_timeout_seconds) so K8s enforces a hard stop and Prefect reliably records a failed terminal state instead of requeuing.

Great choice — I recommend #3 for your case. Why: it’s self-contained in the Kubernetes Job template for your work pool, requires no code changes, and gives you a reliable “hard kill” regardless of Python’s state. With the right settings, the Prefect 2.x Kubernetes worker will see the pod termination and mark the run failed instead of ghost requeuing. Do this: 1) Update your work pool’s Job manifest template - Ensure no automatic restarts: -

restartPolicy: Never

-

backoffLimit: 0

- Add a livenessProbe that intentionally fails after your max runtime - Add a short

terminationGracePeriodSeconds

so K8s escalates to SIGKILL quickly Example template snippet:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: "{{ name }}-"
  namespace: "{{ namespace }}"
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: "{{ finished_job_ttl }}"
  template:
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 15
      containers:
      - name: prefect-job
        image: "{{ image }}"
        imagePullPolicy: "{{ image_pull_policy }}"
        env: {{ env }}
        # Liveness probe triggers a kill at ~3600s
        livenessProbe:
          initialDelaySeconds: 3600
          periodSeconds: 5
          failureThreshold: 1
          exec:
            command: ["sh", "-c", "exit 1"]
        # optional: resources, volume mounts, etc.
        # resources:
        #   limits:
        #     cpu: "4"
        #     memory: "16Gi"

Notes: - The probe will “fire” once at initialDelaySeconds and kubelet will kill the container on the first failure. - For Jobs, with

restartPolicy: Never

and

backoffLimit: 0

, the Pod will be marked Failed and the Job will fail (no retries). - If you already have other probes, make sure they don’t conflict. 2) Set the worker’s watch to outlast the kill - In your Kubernetes work pool variables, set: -

job_watch_timeout_seconds

> liveness initialDelay + terminationGracePeriod + small buffer - Example: initialDelay 3600 + grace 15 → set

job_watch_timeout_seconds

≈ 3700 - This ensures the worker observes the termination and reports a final failed state. 3) Double-check Prefect retry behavior - Make sure you don’t have flow-level retries or external automations that immediately resubmit runs after a failure. Optional: Add activeDeadlineSeconds too - If you want a belt-and-suspenders cluster-level kill, you can also set:

spec:
  activeDeadlineSeconds: 3620

- Keep the worker watch longer than this value as well. If you’d like, paste your current work pool Job template and your worker variables — I’ll help tailor the exact values so K8s enforces the hard stop and Prefect reliably records failure (no ghost requeues). Refs - Worker code (manifest, deletion, watch): prefect_kubernetes/worker.py - Pod event replication: prefect_kubernetes/events.py - Guides: Kubernetes deployment (2.x), Work pools & workers (2.x), prefect-kubernetes (2.x)

You’re right on all counts. Given your goals and the semantics/observability concerns, I recommend using activeDeadlineSeconds over a liveness probe. Why this is the best fit: - It’s the Kubernetes-native, well-understood “hard wall clock” for Jobs - Pod/Job events clearly show DeadlineExceeded - No code changes needed; just update your work pool’s Job template - It avoids the “this process is unhealthy” confusion caused by a liveness probe that intentionally fails Concrete setup 1) Update your work pool Job manifest template

apiVersion: batch/v1
kind: Job
metadata:
  generateName: "{{ name }}-"
  namespace: "{{ namespace }}"
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 3600            # hard wall clock
  ttlSecondsAfterFinished: "{{ finished_job_ttl }}"
  template:
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 15   # short grace so K8s escalates to SIGKILL if needed
      containers:
      - name: prefect-job
        image: "{{ image }}"
        imagePullPolicy: "{{ image_pull_policy }}"
        env: {{ env }}
        # optional: resources, volumes, etc.
        # resources:
        #   limits:
        #     cpu: "4"
        #     memory: "16Gi"

2) Make the worker watch outlast the K8s kill - In your Kubernetes work pool variables, set: -

job_watch_timeout_seconds

>

activeDeadlineSeconds + terminationGracePeriodSeconds + small buffer

- Example: 3600 + 15 + 60 = 3675 (round to ~3700) - Keep

pod_watch_timeout_seconds

as you prefer. 3) Keep no restarts -

restartPolicy: Never

-

backoffLimit: 0

About the “Kopf observer” crash and ghost requeues - Prefect 2.x does not use Kopf; the Kubernetes worker uses the Kubernetes API watchers. If you’re seeing the worker crash while reporting a failed Job (leading to “ghost” requeues), that points to a bug or version mismatch in the worker/watch path. - To help us pinpoint it, can you share: - Prefect version and prefect-kubernetes version - Your work pool variables (especially job_watch_timeout_seconds, pod_watch_timeout_seconds) - The Job manifest template you’re using - Worker logs from around the DeadlineExceeded event - In parallel, make sure you’re on the latest 2.x for both Prefect and prefect-kubernetes; several watch/error-handling fixes have landed over time. If you want a belt-and-suspenders fallback - Run the heavy/hanging code in a child process and SIGKILL it after N seconds (subprocess watchdog). This gives you a second independent hard-stop even if K8s timing is misconfigured. - But in most cases, activeDeadlineSeconds with a slightly longer worker watch is sufficient and the cleanest operationally. References - Worker impl and manifest handling: prefect_kubernetes/worker.py - Pod event replication: prefect_kubernetes/events.py - Guides: Kubernetes deployment (2.x), Work pools & workers (2.x), prefect-kubernetes (2.x) If you paste your current template + worker vars, I’ll help tune exact values to get a clean DeadlineExceeded failure with no ghost requeues.

Great question. In Prefect 2.x’s Kubernetes worker: - Default job watch: job_watch_timeout_seconds = None (no overall timeout) - Default pod watch: pod_watch_timeout_seconds = 60s What this means in practice: - Pod watch: If a pod doesn’t show up within 60s, the worker stops waiting for the pod, logs diagnostics, and returns a failure (-1). It does not auto-retry the watch. - Job watch: With job_watch_timeout_seconds=None, the worker will monitor the Job indefinitely. There’s no outer “hard” timeout, and the watch stream stays attached; it only reconnects if the API requires it (e.g., 410 resource version). If you set a finite job_watch_timeout_seconds, the worker wraps the entire monitoring in a timeout and returns -1 if the job hasn’t completed by then. Additionally, the stream itself will refresh on that interval. So by default, it does not “always refresh if it times out,” because there is no overall timeout applied to the job watch unless you set one. The pod watch has a 60s timeout, but that’s only for the pod bootstrapping phase. Recommendation with activeDeadlineSeconds - Keep using activeDeadlineSeconds for the hard kill. - Set job_watch_timeout_seconds to a bit longer than activeDeadlineSeconds + terminationGracePeriodSeconds + small buffer so the worker is still watching when K8s kills the pod and can mark the run failed. - Example: 3600 + 15 + 60 ≈ 3700 - Keep pod_watch_timeout_seconds as you prefer (default 60 is fine). If you’re seeing the worker crash or a “ghost requeue” after a K8s-enforced kill, that’s not expected. Please share: - Prefect version + prefect-kubernetes version - Your job template and work pool variables - A snippet of worker logs around the DeadlineExceeded event I can help pinpoint whether there’s a bug in the watch/exit-code path or a config edge case.

Sorry, I encountered an error while processing your request. Please try again.

Thanks for the versions — that’s Prefect 3.x (prefect 3.6.22) with prefect-kubernetes 0.7.7. The guidance still applies with this stack. Direct answers: - Default job watch: job_watch_timeout_seconds = None (no overall timeout; will wait indefinitely) - Default pod watch: pod_watch_timeout_seconds = 60s (only for the pod-creation phase) “Shouldn’t it always be watching and refresh if it times out?” - By default, there is no “overall timeout,” so there’s nothing to refresh from — the job watch just stays attached indefinitely and reconnects only on API events like 410 Gone. - If you set job_watch_timeout_seconds to a finite value, the worker computes a deadline and: - Passes the remaining time to each watch stream (so the stream refreshes on that interval) - Aborts and returns -1 once the deadline is reached (it does not continue watching after the overall timeout) What I recommend for your setup - Use activeDeadlineSeconds in the Job manifest (correct semantics, clear DeadlineExceeded events) - Keep restartPolicy: Never and backoffLimit: 0 - Set job_watch_timeout_seconds slightly larger than activeDeadlineSeconds + terminationGracePeriodSeconds + a small buffer so the worker is still watching when K8s kills the pod - Example: activeDeadlineSeconds=3600, terminationGracePeriodSeconds=15 → job_watch_timeout_seconds ≈ 3700 - Keep pod_watch_timeout_seconds as you prefer (60s default is fine) Example job template (work pool Advanced template):

apiVersion: batch/v1
kind: Job
metadata:
  generateName: "{{ name }}-"
  namespace: "{{ namespace }}"
spec:
  backoffLimit: 0
  activeDeadlineSeconds: 3600
  ttlSecondsAfterFinished: "{{ finished_job_ttl }}"
  template:
    spec:
      restartPolicy: Never
      terminationGracePeriodSeconds: 15
      containers:
      - name: prefect-job
        image: "{{ image }}"
        imagePullPolicy: "{{ image_pull_policy }}"
        env: {{ env }}
        # resources, volumes, etc. as needed

If you still see a “ghost requeue” or a worker crash after DeadlineExceeded with this config, that’s not expected. Please share: - Worker logs around the time the pod is terminated (DeadlineExceeded) - Your work pool variables (esp. job_watch_timeout_seconds, pod_watch_timeout_seconds) - The Job template you’re using I’m happy to help pinpoint whether this is a bug or a config edge case and get you to a clean failed terminal state.

Totally fair — a fixed wall clock (activeDeadlineSeconds) isn’t viable if you can sit pending for GPU capacity. Here are options that avoid a global time cap but still give you a “last-resort” hard stop when the workload is actually hung. I’ll list them in the order I’d recommend for your case. Recommended: progress-based liveness (no hard wall clock) - Idea: kill only when there’s no progress for N minutes, not after N minutes total. - How: add a livenessProbe that checks a “progress heartbeat” the workload updates as it runs. If the heartbeat stops advancing for a window (e.g., no update in 30m), kubelet kills the container and the Job fails. Long queueing while Pending is unaffected; long-but-healthy runs keep updating the heartbeat and won’t be killed. - Minimal changes: - Add this to your Job template:

livenessProbe:
      exec:
        command:
          - sh
          - -c
          - |
            f=/tmp/progress.heartbeat
            [ -f "$f" ] || exit 1
            now=$(date +%s); then=$(stat -c %Y "$f"); [ $((now-then)) -lt ${HEARTBEAT_MAX_AGE:-1800} ] || exit 1
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 1

- In your code, update the heartbeat when you make progress (task boundaries, batches, epochs, etc.). Example:

from pathlib import Path
    import time

    def touch_heartbeat(path="/tmp/progress.heartbeat"):
        Path(path).touch(exist_ok=True)

    # call this at start and then on each unit of progress
    touch_heartbeat()
    for step in steps:
        do_work(step)
        touch_heartbeat()

- Why this works: - No kill while Pending (probe runs only when the container is running) - No global wall-clock cutoff - Kills only when the process actually stops progressing - Caveat: if your workload legitimately runs for very long inside a single opaque call that cannot emit heartbeats (e.g., a C/CUDA call that doesn’t return for hours but is healthy), choose a large heartbeat window that reflects your worst-case “no-output” periods. Alternative (code-only, strong isolation): wrap the risky part in a child process and hard-kill on stall - Idea: run the CUDA/C-extension step in a subprocess and SIGKILL the process group if it doesn’t finish in M minutes. This only bounds the risky section; the overall job can run as long as needed. - Example:

import os, signal, subprocess, time

  def run_with_hard_timeout(cmd, timeout_s):
    # new process group so we can kill all children
    p = subprocess.Popen(cmd, preexec_fn=os.setsid)
    try:
      p.wait(timeout=timeout_s)
    except subprocess.TimeoutExpired:
      os.killpg(p.pid, signal.SIGKILL)
      raise RuntimeError(f"Hard timeout after {timeout_s}s")

  # in your task/flow:
  run_with_hard_timeout(["python", "run_my_cuda_step.py"], timeout_s=7200)

- Pro: very deterministic “last resort” kill; unaffected by Python GIL in the child. - Con: requires wrapping each risky section. Infra-heavy but zero app changes: sidecar watchdog that kills on stale progress - Run a lightweight sidecar that watches a progress file or metrics endpoint and deletes the pod if stale for N minutes. Use

shareProcessNamespace: true

if you want the sidecar to signal the main container directly, or just have it call the K8s API to delete the pod. - Pro: no changes inside your flow code entrypoint (if you can surface a progress signal by file or metrics). - Con: more moving parts than a simple liveness probe. What I do not recommend for your scenario - Global activeDeadlineSeconds: you already identified why it’s not viable (counts from Job creation; penalizes long queueing). - “Always-fail” liveness probes: agreed — misleading and not actionable in events. Tie-in with Prefect - When the pod is killed by kubelet (due to liveness failure), prefect-kubernetes will observe the failure and the run will be marked failed. You do not need a Prefect-level timeout.

- If you also want a control-plane kill lever, you can add a Prefect Automation to cancel runs after an upper-bound SLA. Cancellation triggers the worker to delete the Job (hard kill). Just be aware this “run duration” includes time while the run is marked Running, which may include some scheduling time in K8s. If you want, share: - How often you can emit “progress” signals (per batch? per epoch? per file?) - Your tolerance for “no-progress” windows (e.g., 30m? 2h?) - Whether you prefer minimal code change or infra-only I can then give you a ready-to-paste Job template and the smallest code snippet (or a tiny wrapper script) that fits your workload’s shape. For background on probes and semantics, see Kubernetes docs: K8s probes guide.

I can’t open a GitHub issue directly from here, but I’m happy to do one of the following: - Post a fully drafted issue for you to copy/paste into GitHub (recommended), or - Create a GitHub Discussion on your behalf to get maintainer eyes quickly, then convert to an issue if confirmed (I can do this immediately if you prefer) Below is a ready-to-file issue draft for Prefect 3.x in the prefect-kubernetes repo. If you give me the missing logs/repro details, I’ll tighten it up further. Otherwise, you can paste this as-is and edit inline. Direct link to file a new issue: - Open a new prefect-kubernetes issue Suggested title

Kubernetes worker may fail to report final state when Job is killed by K8s (activeDeadlineSeconds/forced kill), leading to “ghost” requeue on EKS 1.33 (Prefect 3.6.22 / prefect-kubernetes 0.7.7)

Issue body (copy/paste) ``` ### Summary On EKS 1.33 using Prefect 3.6.22 and prefect-kubernetes 0.7.7, when a Kubernetes Job is terminated by the cluster (e.g., activeDeadlineSeconds / forced termination of a hung native call), the Prefect Kubernetes worker intermittently fails while reporting the failure, and the flow run gets “ghost” requeued (a new run is created/queued even though the K8s Job has failed). Note: Despite earlier references to “Kopf,” this setup does not use Kopf; it’s the Prefect Kubernetes worker watching Jobs/Pods. ### Environment - Prefect: 3.6.22 - prefect-kubernetes: 0.7.7 - prefect-aws: 0.7.6 - prefect-docker: 0.7.1 - Kubernetes: EKS 1.33 - Work pool: Kubernetes worker ### Configuration - Work pool variables: - `job_watch_timeout_seconds`: [please fill; e.g., None or ~3700] - `pod_watch_timeout_seconds`: [please fill; default 60?] - Job template (relevant): -

restartPolicy: Never

-

backoffLimit: 0

- `activeDeadlineSeconds`: [please fill if used] - `terminationGracePeriodSeconds`: [please fill] - Any retries/automations at the flow/deployment level: [please fill] ### Steps to reproduce 1. Create a Kubernetes work pool and deployment using the above versions. 2. Configure a Job manifest that will be hard-killed by the cluster (e.g., set

activeDeadlineSeconds

), or craft a workload that hangs in a native/CUDA call (uninterruptible by Python). 3. Start a flow run that triggers this Job. 4. Observe that when the pod is terminated by K8s, the Prefect Kubernetes worker attempts to report the failure and sometimes crashes or fails to set a final failed state, and the flow run appears to be re-queued (“ghost” run). ### Expected behavior - When Kubernetes terminates the pod (e.g.,

DeadlineExceeded

, OOMKilled, node eviction), the worker should capture the event, return a non-zero exit code (or -1), and the Prefect run should enter a terminal failed state with a clear reason. - No “ghost” requeue should occur unless explicitly configured via retries/automations. ### Actual behavior - After the K8s-enforced kill, the worker sometimes fails/crashes in the reporting path. - The Prefect run ends up re-queued (a new run appears) even though the underlying K8s Job has failed. ### Logs and diagnostics Please see/attach: - Worker logs around the kill event (DeadlineExceeded / pod termination) - Any stack traces from the worker - Prefect flow run timeline and state transitions - Relevant K8s events from the Job/Pod (e.g.,

kubectl describe job/pod ...

) ### Additional context - Cluster has GPU scheduling constraints: the Job may sit Pending for a long time (so a global

activeDeadlineSeconds

isn’t always viable). - A progress-based liveness strategy (kill on lack of progress) was considered, but the current report is focused on worker behavior when the cluster terminates the pod. - If there’s a more appropriate repo (core vs. prefect-kubernetes) based on the call path, maintainers please advise and I can refile. ### Workarounds tried

- Setting

job_watch_timeout_seconds

to exceed

activeDeadlineSeconds + terminationGracePeriodSeconds + buffer

to keep the worker watching through the kill. - Ensuring

restartPolicy: Never

and

backoffLimit: 0

. - Verifying no flow-level retries/automations causing re-submission. ``` If you’d like, I can instead create a GitHub Discussion now summarizing this and loop in maintainers, then convert to an issue once we have the key logs. Just say “yes, create a discussion,” and I’ll post it and share the link here. Also, if you can paste: - Your work pool variables (job_watch_timeout_seconds, pod_watch_timeout_seconds) - Your Job template - A small snippet of the worker logs around the termination …I’ll incorporate them into the draft before you file it.