Yes — for Prefect 3.7 workers on Kubernetes, the main mechanism is to let Kubernetes send

SIGTERM

, and give the worker enough termination grace period to stop taking new work and finish what it’s already running. What happens on shutdown - Prefect workers handle

SIGTERM

as a graceful shutdown signal - they stop accepting/polling for new work - in-flight work is allowed to continue until completion - if Kubernetes kills the pod before that finishes, the run/task is interrupted So the practical answer is: configure Kubernetes termination properly, because Prefect itself does not currently expose a separate “drain but stay alive” mode for workers. Recommended Kubernetes setup

yaml
spec:
  terminationGracePeriodSeconds: 300
  containers:
    - name: prefect-worker
      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - sleep 10

Why this helps -

terminationGracePeriodSeconds

gives the worker time to finish ongoing tasks - the

preStop

delay gives the pod a moment to be removed from service / settle before full termination - once

SIGTERM

is delivered, the worker should stop taking new work Important caveat for background task workers Since you’re specifically using background task workers, the grace period needs to be at least as long as the longest task you’re willing to let finish during rollout/scale-down. If your tasks can run for 5–10 minutes, set

terminationGracePeriodSeconds

accordingly. If the grace period is shorter, Kubernetes will eventually send

SIGKILL

, and those tasks will be cut off. Operational recommendations 1. Increase

terminationGracePeriodSeconds

to match realistic task durations 2. Use rolling updates with

maxUnavailable: 0

if you want no capacity drop during deploys 3. Scale down slowly so remaining workers can absorb queue load 4. Keep more than one worker replica so one pod can drain while others continue consuming Example rollout strategy:

yaml
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

What Prefect does not currently provide There does not appear to be a dedicated worker “drain mode” like: - stop polling now - stay alive indefinitely until current work finishes - then exit on command So graceful shutdown is primarily controlled by: - signal handling - Kubernetes pod termination settings Extra note for your setup Because you’re autoscaling workers from queue depth, be careful that the autoscaler doesn’t terminate pods too aggressively while they still have long-running tasks. If possible, base scale-down behavior on more than queue size alone, or add a long stabilization window. Relevant docs / references - Prefect workers docs: Workers - Prefect task workers docs: Task workers - Kubernetes pod termination docs: Pod termination lifecycle If you want, I can also help you draft a Kubernetes Deployment spec for your Prefect background task workers with sane rolling-update and graceful-shutdown settings.

Yes — that is the intended behavior. If your Prefect background task worker pod receives

SIGTERM

during Kubernetes scale-down or rollout, then with a long enough `terminationGracePeriodSeconds`: - the worker should stop taking new

.delay()

work - it should continue running the tasks already in flight - once the in-progress tasks finish, the worker can exit cleanly - if everything finishes before the 10-minute grace period, the pod stops normally - if something is still running after 10 minutes, Kubernetes will force-kill it So for your example, 10 minutes is reasonable if all currently running tasks truly complete in under 10 minutes. Important nuance This is graceful shutdown behavior for the worker process. It is not a separate explicit “drain mode” API you turn on. The behavior depends on: - Kubernetes sending

SIGTERM

- Prefect worker handling shutdown correctly - your grace period being longer than the longest in-flight task What I’d recommend in practice Set:

yaml
terminationGracePeriodSeconds: 600

and use a rolling strategy like:

yaml
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

That way: - new worker pod comes up - old worker pod gets terminated - old pod stops taking new tasks - old pod finishes what it already started - old pod exits One thing to be careful about There can be a small race at shutdown time: a worker may already have polled / reserved work just before termination starts. So I would treat this as effectively stops accepting new work, not as a mathematically perfect zero-race drain barrier. If you want to be extra safe, give yourself a little buffer beyond the longest task runtime — for example 12 minutes instead of exactly 10. Docs - Prefect task workers - Prefect workers - Kubernetes pod termination If you want, I can help you verify whether your current worker type is the task worker path specifically, and suggest the exact Kubernetes manifest changes for that deployment.

You’re right — thanks for catching that. For background task workers (

TaskWorker

handling

.delay()

), a

SIGTERM

does not currently behave like a true drain-until-complete shutdown in the same way people expect from flow workers. What happens is closer to: -

SIGTERM

triggers shutdown - the task worker raises

StopTaskWorker

- the subscription loop stops - in-flight background tasks may be interrupted, not guaranteed to finish So your statement is the important one: The “finish in-flight work before exit” behavior should not be assumed for background task workers. That means a Kubernetes

terminationGracePeriodSeconds: 600

by itself is not enough to guarantee that tasks started via

.delay()

will complete before the pod exits. Practical implication If you roll or scale down a background task worker pod: - it should stop taking new work - but currently it may also interrupt work it is already processing - so graceful drain semantics are not guaranteed for background tasks What to do instead Today, safer approaches are: 1. Avoid scaling down active task-worker pods abruptly - use conservative scale-down behavior - add a long cooldown/stabilization window in your autoscaler 2. Roll out new workers before terminating old ones - keep old pods alive while reducing the chance they receive new work - but note this still doesn’t guarantee current tasks survive termination 3. Design background tasks to be idempotent / retry-safe - assume a worker can disappear mid-task - ensure reruns are safe 4. Prefer flow/task execution patterns with stronger lifecycle guarantees if shutdown safety is critical - depending on your architecture, this may mean moving some work out of background task workers 5. Avoid deploying code updates directly onto busy background task workers unless interruption is acceptable Bottom line For Prefect 3.7 background task workers: - No, you cannot rely on SIGTERM + grace period to let in-flight

.delay()

tasks finish automatically - the worker shutdown path can interrupt them I appreciate the correction. If you want, I can help with either of these next: - a safe rollout strategy for Prefect background task workers on Kubernetes despite this limitation - or help check whether this behavior has already been discussed/fixed upstream in GitHub issues/discussions.

Yes — if the worker dies after ACKing the background task, Prefect does not automatically re-schedule that in-flight

.delay()

task for you. So if you want re-execution after worker death, the practical pattern is: mark stuck task runs as failed/crashed, and rely on task retries What happens today There are two cases: 1. Worker dies before ACK - the task can be re-queued automatically 2. Worker dies after ACK, while executing - the task run can be left stuck in a non-final state like

Pending

or

Running

- it is not automatically re-scheduled just because the worker disappeared So for your idempotent tasks, the recovery mechanism needs to be something like: - detect stale/stuck task runs - transition them to

Crashed

or

Failed

- let Prefect retries create another attempt Recommended approach 1. Configure retries on the background task itself

python
from prefect import task

@task(
    retries=3,
    retry_delay_seconds=60,
)
async def my_task(x):
    ...

2. Run a watchdog/monitor that finds task runs stuck in

Running

or

Pending

beyond an expected threshold 3. Set those task runs to

Crashed

via the Prefect API That’s the key step: retries only happen once the task run reaches a retryable failed/crashed state. If it stays stuck in

Running

, nothing else happens. How to implement the watchdog A simple pattern is a periodic job / CronJob that: - queries task runs in

RUNNING

or

PENDING

- filters ones older than some timeout - marks them

Crashed

Pseudo-code:

python
import asyncio
from datetime import datetime, timedelta, timezone

from prefect.client.orchestration import get_client
from prefect.states import Crashed

STALE_AFTER = timedelta(minutes=15)

async def recover_stuck_task_runs():
    cutoff = datetime.now(timezone.utc) - STALE_AFTER

    async with get_client() as client:
        response = await <http://client._client.post|client._client.post>(
            "/task_runs/filter",
            json={
                "task_runs": {
                    "state_type": {"any_": ["RUNNING", "PENDING"]},
                    "state_timestamp": {"before_": cutoff.isoformat()},
                }
            },
        )
        stuck = response.json()

        for tr in stuck:
            await client.set_task_run_state(
                task_run_id=tr["id"],
                state=Crashed(message="Marked stale after worker interruption"),
            )

asyncio.run(recover_stuck_task_runs())

You’d run that every few minutes in Kubernetes as a CronJob. Important caveats - choose the stale threshold to be longer than your longest legitimate task runtime - otherwise you may crash healthy long-running tasks - if a task process is actually still running but not updating state, your watchdog could cause duplicate execution on retry - since your tasks are idempotent, that may be acceptable, but it’s still important to plan for it Stronger safety pattern If duplicate execution matters at all, add your own external idempotency guard, for example: - a database row lock - unique work item ID with processed-state table - compare-and-set before side effects That way, if a retried task starts after a stale one partially ran, side effects are still safe. Alternative option If you control the worker implementation, you could build a custom wrapper around

TaskWorker

to improve shutdown behavior, but out of the box in Prefect 3.7, I would not count on that. Bottom line To ensure interrupted background tasks get run again: - set

retries

on the task - detect stale `Running`/`Pending` task runs - mark them `Crashed`/`Failed` - let Prefect retry them Relevant docs - Task workers: Task workers - Tasks and retries: Tasks - Kubernetes CronJob: Kubernetes CronJob

If you want, I can help you write a real Prefect 3.7-compatible watchdog script using the public client methods instead of the rough pseudo-code above.