You’re right that when Kubernetes force-terminates pods (activeDeadlineSeconds, node preemption, eviction), Prefect can’t always update final state before the process is killed. On spot/preemptible nodes this happens more often, so you want your setup to favor graceful termination, rescheduling, and idempotent recovery. Below are practical, Prefect 2.x-focused recommendations that we’ve seen work well on spot. High-level approach - Run Prefect’s Kubernetes workers on stable (on‑demand) nodes; let only the flow-run pods land on spot. If a worker is evicted, it stops polling and you lose orchestration until it’s back. - Prefer Prefect-level timeouts and retries over hard Kubernetes deadlines. activeDeadlineSeconds tends to cut the process off before Prefect can update state, which looks “fragile” from the UI/API. - Persist results to durable storage and design tasks to be idempotent so restarts/reschedules are safe. Kubernetes work pool and job template - Use a Kubernetes work pool with workers (not the legacy agent + KubernetesJob infrastructure) in Prefect 2.x. - Base job template essentials for spot:

spec:
    backoffLimit: 0                # Prefer Prefect retries over Kubernetes restarts
    ttlSecondsAfterFinished: 3600  # Garbage collect finished jobs
    template:
      spec:
        restartPolicy: Never       # Avoid restart loops inside a single Job attempt
        terminationGracePeriodSeconds: 30-120
        containers:
        - name: prefect
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

Notes: - backoffLimit: 0 avoids K8s-driven restarts that can race with Prefect’s own retry/reschedule logic. If you do want K8s retries, keep it low (e.g., 1-2) and ensure tasks are idempotent to avoid repeated side effects. - restartPolicy: Never is important for clear, single-attempt semantics. - terminationGracePeriodSeconds should be long enough for Prefect to gracefully handle SIGTERM and emit logs/state before SIGKILL. - Avoid or set a very generous activeDeadlineSeconds; use Prefect’s flow/task timeouts to control runtime instead. Worker setup - Keep workers on on-demand nodes (nodeSelector/affinity) so orchestration remains steady if spot nodes churn. - Start the worker against your Kubernetes work pool:

prefect work-pool create "k8s-spot" --type kubernetes
  prefect worker start --pool "k8s-spot"

- If your cluster restricts access to cluster UID discovery, set a fixed cluster identifier on the worker:

PREFECT_KUBERNETES_CLUSTER_UID=your-cluster-id

Flow/task design for resilience - Persist results and use durable storage so retries/reschedules can recover: - For S3, GCS, etc., configure a filesystem block and set

persist_result=True

on flows/tasks. - If using local storage in-cluster, mount a PVC and set

PREFECT_LOCAL_STORAGE_PATH

to that mount so results survive pod eviction. - Add timeouts and retries at the Prefect level:

from prefect import flow, task
  from datetime import timedelta

  @task(retries=3, retry_delay_seconds=10, timeout_seconds=600, persist_result=True)
  def do_work(x): ...

  @flow(retries=2, retry_delay_seconds=30, timeout_seconds=3600, persist_result=True)
  def my_flow(...):
      ...

- Use task caching (

cache_key_fn=task_input_hash

) for expensive, repeatable steps. - Make tasks idempotent: - Use upserts/ON CONFLICT DO NOTHING for DB writes. - Use idempotency keys for external APIs. - Separate side effects from compute where possible (compute → persist artifact → side effect reads artifact). Minimize “hard kills” - Prefer Prefect’s

timeout_seconds

over Job

activeDeadlineSeconds

. If you must use activeDeadlineSeconds (org policy), set it high enough that Prefect’s own timeout will fire first and be logged.

- Ensure

terminationGracePeriodSeconds

is comfortably larger than any cleanup you need (close connections, persist last checkpoints). Scheduling and spot-specific cluster tips - Use a small baseline of on-demand nodes so there’s capacity to reschedule quickly when spot nodes disappear. - Set accurate resource requests to reduce eviction/oomkill risk (under-requesting CPU/memory on spot nodes increases noisy evictions). - In AWS/GCP, deploy the provider’s node termination/preemption handler (e.g., AWS Node Termination Handler) to drain nodes and send SIGTERM earlier; this gives Prefect time to record state and reschedule. Automations and recovery - Create Prefect Cloud automations to react to churn: - On Flow Run Failed/Crash: Resubmit flow run (limit total retries). - On Flow Run Late: Notify and/or resubmit. - This adds a second safety net on top of worker rescheduling and task/flow retries. What to avoid - Relying solely on Kubernetes restarts (high backoffLimit, restartPolicy OnFailure) without Prefect-level retries and result persistence—this makes recovery opaque to Prefect and can look like “lost” state. - Aggressive activeDeadlineSeconds that routinely outpace Prefect timeouts—this yields CRASHED/unknown terminal states because the process is killed before it can report. A simple “spot-ready” baseline 1) Worker on on-demand nodes: - A Deployment with nodeSelector for on-demand nodes, pointing at your “k8s-spot” work pool. 2) Work pool base job template: - backoffLimit: 0, restartPolicy: Never, terminationGracePeriodSeconds: 60, ttlSecondsAfterFinished: 3600. 3) Storage: - Use S3/GCS (recommended) or a PVC for

PREFECT_LOCAL_STORAGE_PATH

. - Set

persist_result=True

on flows/tasks that need recovery. 4) Flow/task config: - retries + timeout_seconds everywhere appropriate; idempotent side effects. 5) Automations: - Resubmit on Failed/Crash with a sensible cap. If you share: - Whether you’re on workers or the legacy agent + KubernetesJob block - Your current base job template (esp. backoffLimit, restartPolicy, activeDeadlineSeconds) - Where you’re storing results (S3/GCS/PVC/ephemeral) …I can tailor this to your exact setup and help tighten the configuration to reduce the “fragile state” behavior you’re seeing on spot nodes.

Thanks for the detailed logs — they’re really helpful. What you’re seeing breaks down into two separate (but related) issues: 1) Thundering-herd/race during submission because the API/DB is slow under load - Multiple workers race to claim/submit the same run → “Aborted submission … cannot transition to PENDING” messages. - This is mostly benign (one of them will win), but it’s noisy and wastes cycles when the server is struggling. 2) Server-side 500s during run startup - The runner is trying to fetch a block type by slug ‘minio-credentials’ and the API returns HTTP 500. - Healthy behavior would be a fast 200 (found) or 404 (missing). A 500 points to API or database pressure, or a bad block registration/migration on the server. Below are concrete steps to stabilize this in Prefect 2.x, especially in a spot-heavy cluster. Immediate mitigations (no code changes) - Reduce worker polling contention - Run fewer workers per work queue. Prefer a smaller number of “beefier” workers over many small ones on the same queue. - Stagger and slow polling a bit to reduce collisions: - Set env on each worker:

PREFECT_WORKER_QUERY_SECONDS=20
      PREFECT_WORKER_PREFETCH_SECONDS=0
      PREFECT_WORKER_HEARTBEAT_SECONDS=30

Prefetch=0 means “only pick runs that are ready right now”, which helps avoid multiple workers fighting over near-future runs. - Use work queue concurrency limits to align with cluster capacity so the server doesn’t try to hand out more runs than you can actually execute. - Keep workers on on‑demand nodes, let only flow-run pods land on spot - If workers churn with spot nodes, the system floods the server with reconnects and re-claims. - Give the server real headroom - API: Add CPU/memory, and scale replicas of the API Deployment/StatefulSet if you’ve split components. - DB: Use an external Postgres with sufficient CPU/IOPS. Add PgBouncer in transaction pooling mode if you have many workers. Ensure autovacuum is keeping up (states and flow_runs churn a lot under load). - Check server logs around the 500s to confirm if it’s DB timeouts, connection pool exhaustion, or a schema issue:

kubectl logs deploy/<your-prefect-server-api> -n <ns> --since=30m

- If you manage DB yourself, a quick health check is connections/locks/slow queries during the spike. Fix the 500 on /api/block_types/slug/minio-credentials - Verify the block type actually exists and is registered on the server - If this is a custom/community block, ensure the server process has the package installed and it’s registered:

# On the server image/pod
    pip install <package-that-defines-your-minio-block>
    prefect block register -m <module_with_block_definitions>

- If you previously upgraded/downgraded Prefect, double-check migrations have run to completion for your server DB (look for migration orion/applied migration logs). A missing column/index can turn a simple lookup into a 500. - If load is high and this request happens early in run startup, consider temporarily avoiding a block call for MinIO: - Inject credentials via env vars (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_ENDPOINT_URL) and read them directly in your code. This removes the early API dependency for block retrieval until the server is stable again. Kubernetes job template for spot (recap) - Use these to make evictions less painful and give Prefect time to update state:

spec:
    backoffLimit: 0
    ttlSecondsAfterFinished: 3600
    template:
      spec:
        restartPolicy: Never
        terminationGracePeriodSeconds: 60

- Prefer Prefect-level timeouts on flows/tasks over activeDeadlineSeconds (hard kills → missing final state). Flow/task settings - Add retries and timeouts everywhere appropriate and persist results to durable storage (S3/GCS or a PVC) so resubmissions can recover cleanly: ```

@task(retries=3, retry_delay_seconds=10, timeout_seconds=600, persist_result=True) def step(...): ... @flow(retries=2, retry_delay_seconds=30, timeout_seconds=3600, persist_result=True) def my_flow(...): ...

- If you’re using local storage, mount a PVC and set:

PREFECT_LOCAL_STORAGE_PATH=/mnt/prefect-results ``` Why the duplicate “submitting flow run” spam happens - With several workers polling the same queue at the same time (and a slow API/DB), more than one can try to submit the same run before the state change is fully processed. The losers abort with “PENDING → PENDING not allowed.” This is noisy but expected under contention. Reducing worker count per queue, increasing query interval, and turning prefetch to 0 typically quiets it down. If you can share a bit more, I can tailor exact knobs: - Prefect 2.x version you’re on - How Prefect Server is deployed (Helm chart? single Deployment?) and resources - Postgres flavor/version and resources; whether PgBouncer is in place - Number of workers and how many work queues they share - Whether “minio-credentials” is a custom/community block (module name), and whether you’ve registered it on the server In the meantime, the fastest path to stop the immediate crashes: 1) Install/register the block type on the server (or bypass it with env vars temporarily). 2) Reduce worker contention (fewer workers per queue, PREFECT_WORKER_PREFETCH_SECONDS=0, PREFECT_WORKER_QUERY_SECONDS≈20–30). 3) Give the API/DB more headroom (scale up, add PgBouncer, check for slow queries/autovacuum lag).