Short answer: yes—but you’ll want to store the DB credentials in a Kubernetes Secret and restart the Server pods when the Secret changes. Prefect reads database settings at process start; it won’t hot-reload a new password without a pod restart. A few key points - In Prefect 3.x, use

PREFECT_SERVER_DATABASE_CONNECTION_URL

(the

PREFECT_API_*

prefix was for 2.x). If you’re on 3.x, please switch to the

PREFECT_SERVER_*

names. - You can set a single connection URL or set individual fields (

PREFECT_SERVER_DATABASE_DRIVER

,

HOST

,

PORT

,

USER

,

NAME

,

PASSWORD

). Using individual vars lets you rotate just the password secret. - To pick up a new value (URL or password), restart the API pods. Typical setup on EKS with the Helm chart 1) Put your DB URL (or password) in a Kubernetes Secret

apiVersion: v1
kind: Secret
metadata:
  name: prefect-db
type: Opaque
stringData:
  url: <postgresql+asyncpg://user:password@host:5432/dbname>
  # or, if you prefer individual fields:
  password: super-secret-password

2) Reference that Secret in the Helm chart values so the server pods get the env var - The chart supports adding env vars to the server container via values.yaml; the exact key path depends on the chart version. Common patterns are an

extraEnv

list on the server/api container. - Example (adjust the path to match your chart’s values—happy to help if you share your chart version or current values.yaml):

server:
  api:
    # add the env var from a Secret
    extraEnv:
      - name: PREFECT_SERVER_DATABASE_CONNECTION_URL
        valueFrom:
          secretKeyRef:
            name: prefect-db
            key: url

    # or, if you are using individual fields instead of a full URL:
    extraEnv:
      - name: PREFECT_SERVER_DATABASE_DRIVER
        value: postgresql+asyncpg
      - name: PREFECT_SERVER_DATABASE_HOST
        value: <http://mydb.example.rds.amazonaws.com|mydb.example.rds.amazonaws.com>
      - name: PREFECT_SERVER_DATABASE_PORT
        value: "5432"
      - name: PREFECT_SERVER_DATABASE_USER
        value: myuser
      - name: PREFECT_SERVER_DATABASE_NAME
        value: mydb
      - name: PREFECT_SERVER_DATABASE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: prefect-db
            key: password

If you’re not sure about your chart’s exact values keys, you can check the chart here: - prefect-helm GitHub repo - Prefect docs: Helm deployment 3) Trigger pod restarts when the Secret changes Options: - Simple/manual: after rotating the Secret, restart the deployment -

kubectl rollout restart deploy/<your-prefect-server-api-deployment>

- Or

helm upgrade --reuse-values <release> prefect/prefect-server

(any change that updates pod template triggers a rollout) - Automated: use a controller like Stakater Reloader to watch Secrets and restart pods when they change; most charts allow adding

podAnnotations

so Reloader can watch them. For example:

server:
  api:
    podAnnotations:
      <http://reloader.stakater.com/auto|reloader.stakater.com/auto>: "true"

- If you use External Secrets Operator (ESO) to pull creds from AWS Secrets Manager, set its refresh interval and use either Reloader or a small CronJob that runs

kubectl rollout restart

after secret sync. Notes and caveats - Existing DB connections remain valid even after a password rotation, but new connections will fail until pods are restarted with the new password—so it’s best to restart promptly after rotation. - If you’re considering AWS RDS IAM auth tokens: those tokens are short-lived and would still require a mechanism to refresh or restart pods. Prefect Server doesn’t generate IAM tokens itself; using a static password in a Secret plus an automated pod restart is usually simpler. Helpful references - Prefect Helm guide: docs-3.prefect.io/v3/advanced/server-helm

- Settings reference (see PREFECT_SERVER_DATABASE_*): docs-3.prefect.io/v3/api-ref/settings-ref - Helm chart repo: github.com/PrefectHQ/prefect-helm If you can share: - Your Prefect version (2.x vs 3.x) - The prefect-helm chart version - A snippet of your current values.yaml …I can give you the exact values.yaml keys for your chart version and the cleanest way to wire in Secret-driven rotation with automatic restarts.

Great question — here’s what that means in practice. What happens when you rotate the Postgres password - Prefect Server (3.x) creates a SQLAlchemy engine and a connection pool at process start using the env vars you set (e.g.,

PREFECT_SERVER_DATABASE_*

). Those connections authenticate once when they are opened. - When you change the user’s password in Postgres: - Active sessions stay alive. Postgres does not kill existing sessions just because the password changed. Those connections keep working until they’re closed or terminated. - New sessions require the new password. Any attempt to establish a new DB connection using the old password will fail with an authentication error. How this shows up in Prefect - Your running Prefect pods keep using their existing pooled connections, so things may look fine immediately after a rotation. - As soon as Prefect needs to open a new DB connection (e.g., because of increased load, a dropped connection, pool recycling, or a DB restart), it will try with whatever credentials it loaded at startup. If the pods were not restarted after the secret was updated, those credentials are stale, and the new connection attempts will fail. - These failures can appear sporadically at first (e.g., when the pool expands or a connection is recycled) and then escalate. By default, SQLAlchemy recycles connections periodically (e.g., pool_recycle ~ 3600s), so you may see more failures as old connections get closed and replaced. Why restart promptly - Restarting the pods right after the Secret is updated forces Prefect to rebuild the SQLAlchemy engine and load the new password, so all new connections authenticate successfully. - If you delay the restart, you’ll get intermittent DB auth errors when new connections are needed. Recommended patterns for your setup (EKS + Helm + ESO) - Keep using ESO to sync the AWS Secret into a Kubernetes Secret. - Automate rollouts on Secret changes so the restart happens as soon as ESO updates the Secret: - Use a tool like Stakater Reloader and annotate the server pods so they roll when the Secret changes. - Or add a small automation (e.g., a CronJob or CI/CD step) that runs a

kubectl rollout restart

of the API/services deployments when ESO updates the Secret. - For zero/near-zero disruption: - Do a rolling restart immediately after the secret update so new pods start with the new password while old pods continue using their existing connections. - An even safer approach is to rotate to a different DB user: create a new user with the new password, update the Secret to use the new user, roll the pods, then remove the old user. This avoids any window where old pods might need to create new connections with an invalid password. Relevant docs - Prefect settings reference (PREFECT_SERVER_DATABASE_*): docs-3.prefect.io/v3/api-ref/settings-ref - Helm deployment guide: docs-3.prefect.io/v3/advanced/server-helm If you’d like, I can suggest exact values.yaml annotations for your chart version (2025.6.4170433) to wire up a Secret-watcher and perform rolling restarts on password changes.

Short answer: not really. Whether you run Prefect Server on EKS or ECS, the process reads DB creds at startup. When the Postgres password rotates, you need to restart the process so new connections use the new password. ECS does not automatically “hot-reload” secrets either. What can make it feel “easier” on ECS is that AWS-native glue is built-in: - ECS task definitions can pull env vars directly from AWS Secrets Manager. - You can wire an EventBridge rule on a Secrets Manager rotation event to trigger a force-new-deployment of your ECS service so tasks restart automatically. But the same level of automation is straightforward on EKS: - Keep using External Secrets Operator to sync the secret. - Add an automatic rollout on Secret changes (e.g., Stakater Reloader or an operator that bumps a checksum annotation) so the Prefect Server pods restart as soon as the secret updates. If your goal is to avoid restarts entirely during password rotations, consider putting an RDS Proxy in front of your Postgres: - Prefect connects to the proxy; the proxy manages/refreshes DB credentials behind the scenes. - Password rotations become transparent to clients; you don’t need to restart Prefect when the DB password changes. - This works regardless of EKS vs ECS and is usually the lowest-ops pattern for frequent rotation. Concrete options - EKS automation (no proxy): - Use a secret-watcher to roll pods on Secret change, e.g.: - Stakater Reloader annotation on the Prefect Server deployment - Or a small Job/Cron/Lambda that runs

kubectl rollout restart

after ESO syncs the secret - ECS automation (no proxy): - Store DB creds in Secrets Manager, enable Secrets Manager rotation for your DB user - EventBridge rule on rotation completion -> Lambda to force a new ECS deployment:

import boto3
    def handler(event, context):
        ecs = boto3.client('ecs')
        ecs.update_service(cluster='my-cluster',
                           service='prefect-server',
                           forceNewDeployment=True)

- Best overall for frequent rotations: - Use RDS Proxy with either: - DB password stored in Secrets Manager and rotated by AWS; proxy updates automatically - Or IAM auth to the proxy (no static password in Prefect) - Prefect uses a stable connection string to the proxy; no app restarts needed on rotations If you’d like, I can: - Share exact values.yaml snippets to wire a Secret-watcher for your helm chart version (2025.6.4170433), or - Sketch an ECS setup with Secrets Manager rotation + EventBridge/Lambda to auto-redeploy, or - Outline an RDS Proxy connection string and settings for Prefect Server.

Short answer: using a ConfigMap won’t avoid restarts. - Kubernetes freezes environment variables at container start. If you set

PREFECT_SERVER_DATABASE_CONNECTION_URL

via `env`/`envFrom` (from a ConfigMap or a Secret), updating that resource does not change the env var inside running pods. - Even if you mount a ConfigMap/Secret as a volume, Prefect Server does not watch files or hot-reload settings. It reads its settings at process start. So you will still need to restart the server pods whenever the DB URL or password changes, regardless of ConfigMap vs Secret. Recommended approach - Keep credentials in a Secret (not a ConfigMap). - Automate pod restarts when the Secret changes: - Use Stakater Reloader to watch the Secret and roll the deployment automatically. Example pod annotation you can add to the Prefect Server API:

server:
      api:
        podAnnotations:
          <http://reloader.stakater.com/auto|reloader.stakater.com/auto>: "true"

- Or trigger a rollout from your pipeline (e.g., after External Secrets Operator syncs the new value). References - Kubernetes ConfigMaps (env vars do not update in-place): https://kubernetes.io/docs/concepts/configuration/configmap/ - Stakater Reloader: https://github.com/stakater/Reloader - Prefect Helm server guide: https://docs-3.prefect.io/v3/advanced/server-helm - Prefect settings reference (PREFECT_SERVER_DATABASE_*): https://docs-3.prefect.io/v3/api-ref/settings-ref

In Prefect 3.x, settings are read from environment variables when the process creates its Settings object—effectively at startup—and then cached for the lifetime of that process. Key behaviors: - Environment variables are not hot-reloaded. Changing an env var in the pod/node after Prefect starts will not change Prefect’s effective settings. - Prefect uses a “current settings context” that holds a Settings instance. Most code resolves values via that context, e.g.,

PREFECT_API_URL.value()

->

get_current_settings()...

. - You can override settings programmatically within a scope using

temporary_settings(...)

, which swaps in a new Settings instance for that scope. This does not re-read OS env; it applies the values you pass. Implications for your server pods: - If you change

PREFECT_SERVER_DATABASE_CONNECTION_URL

(or its password component) via a Secret/ConfigMap update, the running Prefect server will not pick it up until the process restarts and constructs a new Settings instance. - Therefore, you should trigger a rolling restart of the server pods when the secret updates. Docs: - Settings and profiles overview: docs-3.prefect.io/v3/concepts/settings-and-profiles - API reference for settings context and overrides: docs-3.prefect.io/v3/api-ref/python/prefect-settings-context

Short answer: no—Prefect Server won’t fall back to a .pgpass file. Why: - Prefect 3.x Server uses SQLAlchemy with the asyncpg driver (

postgresql+asyncpg

). asyncpg does not use libpq, so it does not read

.pgpass

(or

.pg_service.conf

). It expects credentials to be provided explicitly (URL or discrete fields). - Prefect’s settings system only reads from environment variables, profiles, and config files it knows about; it does not scan

.pgpass

. What you can do instead: - Keep using a Kubernetes Secret (populated by External Secrets Operator) and inject either: - a single

PREFECT_SERVER_DATABASE_CONNECTION_URL

, or - individual fields (

PREFECT_SERVER_DATABASE_PASSWORD

, etc.). Then roll the pods when the Secret changes. - If you must source from a file, use a small wrapper/entrypoint that reads the mounted

.pgpass

, exports the appropriate

PREFECT_SERVER_DATABASE_*

env vars, then execs the server. Example sketch:

#!/usr/bin/env sh
  set -euo pipefail

  # Parse the matching line: host:port:database:user:password
  LINE=$(grep -m1 '^mydbhost:5432:mydb:myuser:' /mnt/pgpass/.pgpass || true)
  PASSWORD=$(printf '%s' "$LINE" | awk -F: '{print $5}')

  export PREFECT_SERVER_DATABASE_DRIVER=postgresql+asyncpg
  export PREFECT_SERVER_DATABASE_HOST=mydbhost
  export PREFECT_SERVER_DATABASE_PORT=5432
  export PREFECT_SERVER_DATABASE_USER=myuser
  export PREFECT_SERVER_DATABASE_NAME=mydb
  export PREFECT_SERVER_DATABASE_PASSWORD="$PASSWORD"

  exec prefect server start

This requires a custom image or command override and a volume mount for the pgpass file. Best-practice options if you rotate credentials: - Continue with ESO + Secret + automatic rollout on change (e.g., Stakater Reloader), since Prefect won’t hot-reload settings. - Consider RDS Proxy so rotations are transparent to clients; Prefect connects to the proxy with a stable configuration. References: - Prefect settings reference (see PREFECT_SERVER_DATABASE_*): docs-3.prefect.io/v3/api-ref/settings-ref

Short answer: your running flows will keep running. Details: - If you roll Prefect Server pods (API/UI/Scheduler) to pick up the new DB secret: - Flow runs already executing (e.g., as Kubernetes Jobs) continue independently of the API. They won’t be interrupted. - While the API is briefly unavailable, state/log updates may be delayed in the UI. Once the API is back, the worker/flow reconciles state and you’ll see the correct terminal status. - New scheduled runs may start a little late if the API is down at their scheduled time; they’ll be picked up once the API is back. - If you roll your Prefect workers: - Existing Kubernetes Jobs keep running to completion; workers are stateless. After restart, a worker will resume monitoring jobs and updating state. - If a job finishes while the API is unreachable, its final state may publish a bit later when connectivity returns; workers tolerate brief outages. By default, workers have a generous health grace period (~7.5 minutes) before being considered unhealthy. Recommendations for secret rotation with minimal impact: - Keep API downtime as short as possible and consider 2+ replicas for the API behind a Service so you can do a rolling restart without losing availability. - Use a PodDisruptionBudget (minAvailable: 1) and RollingUpdate strategy (maxUnavailable: 0) for the API. - Don’t roll workers unless needed; if you do, keep at least one worker available or run multiple replicas so new work is still picked up. - Ensure your flows write results to durable storage (e.g., S3) so outputs aren’t tied to API availability. References: - Workers overview - Worker health checks - States - Self-hosted/HA guidance Quick check: which pods are you planning to roll (Prefect Server only, or workers too), and are your flows running via the Kubernetes worker?

If you roll just the Prefect Server pods, your running flow pod will keep running. Two cases based on what you meant by “query against the Prefect DB”: 1) Typical/expected case (flows don’t talk to the Prefect DB directly) - Your flow code runs inside a Kubernetes Job and is independent of the API. While the API pods roll, the job keeps running. - State/log updates to the API may be delayed during the brief outage. Once the API is back, the worker/flow reconciles and you’ll see the correct state and logs. - If the flow finishes while the API is down, its terminal state will be published once connectivity resumes (the Kubernetes worker reconciles job completion). 2) If your flow truly connects directly to the Prefect PostgreSQL database - Rolling the API pods does not affect the database process itself, so a long-running DB query typically continues uninterrupted. - If you are also rotating the DB password at the database, existing Postgres sessions generally remain valid and keep running; new connections will require the new password. If you forcibly terminate sessions or change auth in a way that invalidates active sessions, the query would be killed and the flow would fail accordingly. - Note: having flows query the Prefect orchestration database directly is not recommended. Flows should use the Prefect API; direct DB access can break invariants and isn’t supported. A few edge considerations - Any flow code that makes live API calls mid-run (e.g., loading a block, creating runs, writing artifacts) could raise if the API is briefly unavailable. Wrap those calls with retries if they’re on the critical path during a rollout. - You may see a temporary gap in UI logs during the restart; they will generally backfill once connectivity returns, but a few lines right at process exit could be lost in a worst case. Recommendations for secret rotation with minimal/no impact - Run 2+ replicas of the Prefect API behind a Service with a rolling update strategy (maxUnavailable: 0) and a PodDisruptionBudget so rollouts don’t cause downtime. - Avoid terminating active DB sessions during password rotation. Update the Secret, roll the server Deployments, and let existing DB sessions drain naturally. - Keep the flow’s result storage durable (e.g., S3) so outputs are not tied to API availability. - If your flows need to query Postgres for application data, use a separate database/user from the Prefect orchestration DB. References - Workers overview - Worker health checks and resilience - States - Self-hosted/HA guidance Quick check: are your flows actually connecting to the Prefect orchestration database, or was that just a hypothetical? If they do, I’d recommend we shift them to use the Prefect API (or a separate application DB) to avoid unintended coupling.

Great question. In your scenario: - Your flow pod keeps running. - The in-flight API request from

get_client()

to the Prefect Server that’s currently executing a long DB query will be interrupted when that server pod is terminated during the rollout. - From the flow’s perspective, that specific client call will raise (connection reset/timeout or a 5xx). Prefect does not automatically retry arbitrary client calls on connection failure; you should add retries around any API calls your code makes. What this means in practice - After rollout, reissuing the same request will hit a new API pod and succeed (assuming the DB is healthy). The original long DB query on the terminated pod won’t complete. - The rest of the flow continues; only that API call fails unless you retry it. Recommendations to minimize interruption - Run 2+ API replicas behind your Service and use a rolling update with maxUnavailable=0 and a PodDisruptionBudget so traffic drains before a pod is killed. A short preStop sleep and adequate terminationGracePeriodSeconds can help in-flight requests finish. - Add retries/timeouts to your client calls in flow code. Example:

from prefect import task
from prefect.client.orchestration import get_client
import httpx

@task(retries=3, retry_delay_seconds=3)
async def call_prefect_api():
    async with get_client(httpx_settings={"timeout": 30.0}) as client:
        # Replace with the API call you need to make
        # return await client.some_operation(...)
        try:
            ...
        except httpx.HTTPError:
            # Let the task-level retries handle transient rollout errors
            raise

- Keep API interactions idempotent where possible so retries are safe (reads/listing artifacts generally are). - Expect a brief gap in UI logs right around the rollout; logs are batched and may be dropped if a batch send coincides with termination. Docs for reference - Self-hosted/HA guidance: Self-hosted Prefect - Client settings (timeouts, retry knobs): Settings reference - Rate limit behavior (the only automatic client retries): Rate limits If you’re rolling server pods to pick up a rotated DB password, this approach (multiple replicas + rolling update) will avoid dropping in-flight calls in most cases. Do you currently have multiple API replicas configured via your Helm values? If you’d like, share your relevant values and I can suggest tweaks for safe rollouts.

You can run multiple Prefect Server API pods behind the same Service so rollouts don’t interrupt client calls. The exact Helm values are chart-version specific, so first check your chart’s defaults, then set replicas and add a PDB. 1) Inspect your chart values (so we use the right keys)

helm show values prefect/prefect-server --version 2025.6.4170433 | less
helm get values <release> -n <namespace> -o yaml

Common keys are either

server.replicaCount

or

api.replicaCount

(the chart has used both patterns across versions). Use whichever exists in your chart. 2) Set 2+ replicas via Helm - If your chart uses

server.*

keys:

helm upgrade <release> prefect/prefect-server \
  -n <namespace> \
  --version 2025.6.4170433 \
  --set server.replicaCount=2

- If it uses

api.*

keys:

helm upgrade <release> prefect/prefect-server \
  -n <namespace> \
  --version 2025.6.4170433 \
  --set api.replicaCount=2

3) Add a PodDisruptionBudget (works regardless of chart keys) using extra objects in your values

extraObjects:
  - apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: prefect-server-pdb
    spec:
      minAvailable: 1
      selector:
        matchLabels:
          <http://app.kubernetes.io/name|app.kubernetes.io/name>: prefect-server
          <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: <release>

Note: adjust labels if your chart uses different ones; most Helm charts set those two labels. 4) Rolling updates and probes - Most chart versions use a Deployment with RollingUpdate by default. If your chart exposes strategy knobs, set maxUnavailable=0:

# Use whichever block your chart exposes (server.strategy or api.strategy)
server:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

- Ensure readiness probes point to

/api/readiness

and liveness to

/api/health

on port 4200 (these are the Server endpoints). The chart generally sets these, but confirm in

helm show values

. Optional for graceful shutdowns - If supported by your chart, set a small preStop and adequate terminationGracePeriodSeconds so inflight requests finish:

server:
  terminationGracePeriodSeconds: 60
  lifecycle:
    preStop:
      exec:
        command: ["sh", "-c", "sleep 5"]

If you want, paste the output of:

helm show values prefect/prefect-server --version 2025.6.4170433 | sed -n '1,200p'

and your

helm get values <release> -n <ns> -o yaml

, and I’ll pinpoint the exact keys for your chart version and provide a drop-in values.yaml. References: - Prefect self-hosted/HA: Self-hosted Prefect - Helm chart docs: Prefect Server Helm guide - Helm repo: prefect-helm - K8s PDB: PodDisruptionBudget

Totally fair question. Multiple API replicas don’t “save” a request on a pod that you actually terminate. They help by keeping the service available while you roll pods and by giving in‑flight requests time to finish on the draining pod. Here’s how it plays out: - Single replica - When you roll the Deployment, Kubernetes sends SIGTERM to the only API pod and, after the grace period, kills it. Any in‑flight request on that pod (e.g., your long artifact filter query) is terminated. Clients see a connection error. - Multiple replicas with proper draining - RollingUpdate with maxUnavailable=0 ensures at least one replica stays Ready throughout. - On each pod being updated, Kubernetes first marks it NotReady (via readiness probe) so it stops receiving new traffic, then waits for terminationGracePeriodSeconds before force-killing it. If you add a small preStop delay, the pod is removed from the Service/endpoints before SIGTERM is sent. - Any in‑flight requests on that draining pod have up to terminationGracePeriodSeconds to complete. New requests go to the other replica(s) so the API stays available. - If your long query completes within the grace period, it won’t be interrupted. If it exceeds the grace period, it will still be terminated. So, replicas + draining don’t magically make a killed process finish; they: - keep the API up during rollouts, and - give ongoing requests time to finish on the pod you’re draining. What you need to avoid interruptions for long queries - Run 2+ API replicas and use a zero‑downtime rollout: - RollingUpdate: maxUnavailable=0, maxSurge=1 - Readiness probe (stops new traffic before termination) - preStop sleep (a few seconds) to let endpoints/LB drain connections - terminationGracePeriodSeconds long enough to cover your longest API call (e.g., 300s if your queries can run up to 5 minutes) - A PodDisruptionBudget with minAvailable: 1 - At the edge (ALB/NLB), enable connection draining/deregistration delay so in‑flight connections are allowed to finish after the pod is removed from the target set. - In your flows, add retries/timeouts to API calls (Prefect doesn’t auto‑retry arbitrary connection failures), and prefer paginated/list endpoints over single massive queries to reduce per‑request duration. Concrete snippets (adapt keys to your chart; some versions use server., others api.): - Replicas and rollout

server:
  replicaCount: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  terminationGracePeriodSeconds: 300
  lifecycle:
    preStop:
      exec:
        command: ["sh", "-c", "sleep 5"]
  readinessProbe:
    httpGet:
      path: /api/readiness
      port: 4200
  livenessProbe:
    httpGet:
      path: /api/health
      port: 4200

- PodDisruptionBudget (via extra objects)

extraObjects:
  - apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: prefect-server-pdb
    spec:
      minAvailable: 1
      selector:
        matchLabels:
          <http://app.kubernetes.io/name|app.kubernetes.io/name>: prefect-server
          <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: <release>

- ALB connection draining (example annotation; adjust for your setup)

service:
  annotations:
    <http://service.beta.kubernetes.io/aws-load-balancer-target-group-attributes|service.beta.kubernetes.io/aws-load-balancer-target-group-attributes>: deregistration_delay.timeout_seconds=300

- Add retries around client calls in your flows

from prefect import task
from prefect.client.orchestration import get_client
import httpx

@task(retries=3, retry_delay_seconds=5)
async def list_artifacts_with_retry(...):
    async with get_client(httpx_settings={"timeout": 60.0}) as client:
        try:
            # Use pagination to avoid one giant query
            # e.g., await client.read_artifacts(filter=..., limit=500, offset=...)
            ...
        except httpx.HTTPError:
            raise  # let task retries handle transient rollout errors

Key takeaways

- With one replica, any rollout will interrupt in‑flight requests on that pod. - With multiple replicas and proper draining, in‑flight requests on the draining pod can finish within your grace window, and new requests continue on the other pods. - For truly long operations, combine HA rollout + longer grace period + client retries + pagination. References - Self-hosted/HA guidance - Prefect Server Helm guide - Kubernetes PodDisruptionBudget - Kubernetes pod termination and preStop If you share your current Helm values for the server Deployment, I can map these settings to the exact keys for your chart version (2025.6.4170433).

With a single Prefect Server pod and terminationGracePeriodSeconds=30: - When the rollout starts, Kubernetes marks the pod NotReady and stops routing new requests to it. It sends SIGTERM and starts the 30s grace timer. - Any in-flight request on that pod (e.g., your long DB query) has up to 30 seconds to finish. If it completes within 30s, it succeeds. If it runs longer, Kubernetes sends SIGKILL at 30s and the request is aborted; clients will see a connection error. - Because there’s only one replica, the API is unavailable for new requests from the moment the pod is drained until the new pod becomes Ready (30s grace + startup/readiness time). During that window, any flow/client calls to the API will fail. - Running flows (e.g., k8s Jobs) keep executing; only their API interactions during the downtime are impacted (state/log updates may error or be delayed; a few log lines might be lost). Once the new server pod is up, workers/flows resume normal communication and reconcile terminal states. If you need uninterrupted long queries, 30s usually isn’t enough. Either: - increase

terminationGracePeriodSeconds

to exceed your longest request, and set your load balancer’s deregistration/drain timeout to match, or - run 2+ replicas with draining so the API stays up during rollouts and in-flight requests can finish on the draining pod. References: - Kubernetes pod termination behavior: Pod termination - Prefect self-hosted/HA guidance: Self-hosted Prefect

Short version: SIGTERM does not “pause and resume” a running DB query. With a single Prefect Server pod, any in‑flight SQLAlchemy query will either finish during the grace window or be aborted when the pod is killed. What actually happens on SIGTERM - The server stops accepting new requests and begins graceful shutdown. - Any in‑flight request handler (including the SQLAlchemy call) keeps running until: - It completes before the pod’s terminationGracePeriodSeconds expires → the request succeeds, and the DB transaction commits/returns normally. - The grace period expires → Kubernetes sends SIGKILL, the process dies, the DB connection is dropped, and Postgres aborts the running statement. Your client sees a failed HTTP request (typically a connection reset/timeout/5xx). No partial writes are committed; the transaction is rolled back. Important implications with 1 replica - There’s no other pod to receive traffic during the rollout; the API is unavailable until the new pod is Ready. - The specific long query that was in flight on the terminated pod is not retried automatically by the server. If your flow retried the API call, it would succeed against the new pod once it’s up. How to reduce interruptions - Increase terminationGracePeriodSeconds to exceed your longest server-side request, so most queries finish before the kill. - Better: run 2+ API replicas with proper draining (readiness probe, preStop, PDB, maxUnavailable=0) so in‑flight requests can complete on the draining pod while new requests go to the other replica(s). - Consider a Postgres statement_timeout set just under your grace period for predictable cancellations. - In your flow code, add retries/timeouts around API calls so a single rollout-induced failure is retried. References - Prefect self-hosted/HA: Self-hosted Prefect - Kubernetes pod termination: Pod termination

Short answer: Prefect’s FastAPI app relies on the ASGI server (Uvicorn) for shutdown. On SIGTERM, it performs a standard graceful shutdown; there’s no special “pause/resume” for DB work. What happens on SIGTERM - The Uvicorn server stops accepting new connections and begins graceful shutdown. - FastAPI lifespan shutdown runs. Prefect uses this to stop background services (scheduler, late-runs, events, etc.) and close resources like database connection pools. - Request-scoped DB sessions (SQLAlchemy/asyncpg) are closed when their request finishes; on shutdown, the engine/pool is torn down. - Any in-flight request: - If it completes before the pod’s terminationGracePeriodSeconds expires, it returns normally and commits as usual. - If it’s still running when the grace period ends, Kubernetes sends SIGKILL; the process dies, the Postgres connection is dropped, and Postgres cancels the running statement. The HTTP request fails; no partial transaction is committed. There isn’t extra application-level signal handling in the Prefect API that would keep a long DB query alive past process termination. It’s the standard FastAPI/Uvicorn behavior. How to reduce impact - Increase terminationGracePeriodSeconds to exceed your longest API call. - Add a small preStop hook (e.g., sleep 5s) so the pod is removed from Service endpoints before SIGTERM, giving it time to drain. - Prefer multiple API replicas with maxUnavailable=0 so new requests continue on healthy pods during a rollout. - For predictability, you can set a Postgres statement_timeout slightly below your grace period so long queries are canceled cleanly before SIGKILL. - In flows, add retries/timeouts around API calls in case a rollout cuts one off. References - Self-hosted/HA guidance: Self-hosted Prefect - Prefect Server Helm overview: Prefect Server Helm - Kubernetes pod termination behavior: Pod termination