Thanks for the detailed write-up — this lines up with what we’ve seen when running Prefect Process workers inside Kubernetes that autoscale. What you’re seeing - Process workers do not “drain” on SIGTERM. When the pod is terminated (e.g., during downscale), the worker shuts down quickly and any in-flight flow runs can be left in RUNNING because the process hosting them is gone. -

terminationGracePeriodSeconds

won’t help if the worker exits immediately on SIGTERM — the grace period is time the kubelet will wait before SIGKILL, not time the worker will actively drain. - This is distinct from the Kubernetes worker: k8s workers submit work as Kubernetes Jobs/Pods and those continue running even if the worker goes away. Docs for context: - Workers overview: Workers in Prefect 3.x - Process worker API: ProcessWorker API - Detecting “zombie” (stuck RUNNING) flows with automations: Detect zombie flows Recommended approaches 1) Prefer the Kubernetes worker in Kubernetes environments - This decouples worker lifetime from flow runs; downscaling the worker will not strand runs. - If you switched to Process to save resources, consider running a small K8s worker (low concurrency) that scales to 0 when idle; submitted Jobs will still complete without the worker present. 2) If you must keep Process workers in K8s, use mitigations - Add a PodDisruptionBudget and annotate worker pods to reduce autoscaler-initiated evictions: - PDB: ensure

minAvailable: 1

for the worker deployment. - Annotation:

<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"

on the worker pod to discourage CA from picking it first. - Add a container lifecycle preStop hook to buy time for short in-flight runs: - preStop runs before SIGTERM is sent; a simple sleep can allow short flows to finish. - Example snippet you can adapt into your Helm values:

lifecycle:
      preStop:
        exec:
          command:
            - /bin/sh
            - -c
            - 'echo "preStop: waiting 90s for active runs..." && sleep 90'
    terminationGracePeriodSeconds: 120

- Note: this only helps if your runs typically finish within that window. The Prefect process worker does not actively drain; it will still exit once SIGTERM arrives. - Wrap the worker in a tiny entrypoint that traps SIGTERM and waits briefly for children:

#!/bin/sh
  set -e
  prefect worker start -p "$PREFECT_WORK_POOL" &
  PID=$!
  trap 'echo "SIGTERM received; waiting 60s for active runs"; sleep 60; kill $PID; wait $PID' TERM
  wait $PID

- This is a best-effort workaround; it doesn’t guarantee graceful draining but can reduce stranded runs if your flows are short. - Configure a “zombie flow” automation so stranded runs don’t remain RUNNING indefinitely: - In the UI, create an Automation → Condition: “Flow run missing heartbeat” (e.g., >2–5 minutes) → Action: “Mark as CRASHED” (or “Retry” if appropriate). - Docs: Detect zombie flows A few clarifying questions so we can suggest exact Helm values: - Which Helm chart and version are you using? - Prefect server and worker versions? (You mentioned

prefecthq/prefect:3.6.1-python3.11-kubernetes

— what were you on before?) - How are you starting the worker in the chart (command/args)? Any custom lifecycle hooks already? - Typical duration of your flows? If they’re consistently <60–90s, a preStop window may be sufficient. If you’re open to it, I’d recommend going back to the Kubernetes worker for EKS — you still keep resource use low (scale-to-zero worker) while avoiding this class of stranded RUNNING states.

If you’d like, share your Helm values for the worker and I’ll propose concrete patches (preStop, PDB, annotations) tailored to your setup.

@Marvin Show me the prestop config for a worker type process

Assuming you’re running a Prefect 3.x worker of type

process

on Kubernetes: you generally don’t need a preStop hook because the worker already handles SIGTERM gracefully. That said, if you want an explicit preStop to initiate shutdown and give it time to drain, here’s a safe example you can drop into your Pod/Deployment spec. Minimal container snippet

containers:
  - name: prefect-worker
    image: prefecthq/prefect:3-latest
    command: ["prefect", "worker", "start", "--pool", "<your-pool>", "--type", "process", "--with-healthcheck"]
    lifecycle:
      preStop:
        exec:
          command:
            - /bin/sh
            - -c
            - |
              echo "preStop: sending SIGTERM to Prefect worker and waiting"
              kill -TERM 1
              sleep 30
    terminationGracePeriodSeconds: 60

Notes: - The preStop sends SIGTERM to PID 1 (the worker) and waits 30s to let Prefect shut down cleanly. Adjust the sleep and terminationGracePeriodSeconds to your needs. - If you omit preStop entirely, Kubernetes will still send SIGTERM on termination and Prefect will shut down cleanly. - Consider

--with-healthcheck

so your readiness/liveness probes can reflect the worker’s state. References: - Workers in Prefect 3.x - Worker healthchecks - Kubernetes infrastructure guide If you’re using Helm, do you want this embedded in a values.yaml override or a patch to a specific chart? I can tailor the snippet accordingly.

Thanks for the detail — and you’re right to flag this. With a Process worker, flows run in child subprocesses. When the worker gets SIGTERM, those children are typically terminated too. There is no built‑in “drain” period or shutdown timeout for Process workers in Prefect 3.x, so “graceful” here means the worker stops cleanly, not that it waits for in‑flight runs to finish. If you want graceful drain on Kubernetes, the pattern is: - Don’t send SIGTERM in preStop. - Pause the pool (and/or queues) so the worker stops picking up new work. - Wait for the worker’s child processes to finish. - Then let Kubernetes deliver SIGTERM. Here’s a preStop you can use for a Process worker that implements that drain-first behavior:

containers:
  - name: prefect-worker
    image: prefecthq/prefect:3-latest
    command: ["prefect", "worker", "start", "--pool", "<your-pool>", "--type", "process", "--with-healthcheck"]
    env:
      # ensure the CLI in preStop can auth to your API
      - name: PREFECT_API_URL
        value: "<your-api-url>"
      - name: PREFECT_API_KEY
        valueFrom:
          secretKeyRef:
            name: prefect-api
            key: api_key
    lifecycle:
      preStop:
        exec:
          command:
            - /bin/sh
            - -c
            - |
              set -e
              POOL="<your-pool>"
              echo "preStop: pausing work pool $POOL"
              prefect work-pool pause "$POOL" || true

              # Optional: if you target specific queues, pause them too
              # prefect work-queue pause "<your-queue>" || true

              echo "preStop: waiting for in-flight subprocesses to finish"
              MAX_WAIT=600   # total seconds to wait
              SLEEP=5
              waited=0
              # Check for child processes of PID 1 (the worker process in the container)
              while pids="$(ps -o pid= --ppid 1)"; [ -n "$pids" ] && [ $waited -lt $MAX_WAIT ]; do
                echo "Still running child processes: $pids"
                sleep $SLEEP
                waited=$((waited+SLEEP))
              done

              if [ -n "$pids" ]; then
                echo "Timeout reached; proceeding with termination"
              else
                echo "No child processes remain; proceeding"
              fi
    terminationGracePeriodSeconds: 90

Why this works: - preStop runs before Kubernetes sends SIGTERM. We use that window to stop new work (pause) and wait for children to exit. - After preStop completes, Kubernetes sends SIGTERM to the worker; by then there should be no in‑flight flow subprocesses left to kill. Alternatives and tips: - If you run on Kubernetes, consider using the Kubernetes or Docker worker types for isolation. They launch each flow in a separate Pod/container so worker termination does not kill in‑flight runs. - If you prefer readiness-based draining, start the worker with

--with-healthcheck

and use a readinessProbe to take the pod out of service before scale-down, then use preStop to wait for children to finish. - There isn’t a worker “drain mode” or shutdown timeout setting in Prefect 3.x today for Process workers. Docs for context: - Workers overview: Workers in Prefect 3.x - Healthchecks: Worker healthchecks - Kubernetes infra guide: Kubernetes infrastructure guide If you can share: - How you’re running the worker (exact command/image), and - Whether this is on Kubernetes, Docker, or systemd I can tailor the preStop and probes to your setup (or provide a Helm values.yaml override).