Short answer - Yes. If your Prefect Server API is saturated, client calls from workers/flows can time out. Prefect will retry transient API errors, but after retries are exhausted you’ll see API errors in logs and runs may get stuck, be marked late, or end in crashed/failed depending on where the timeout occurred. - You can measure load by watching API latency/error rates (e.g., hitting /api/health), infra metrics (CPU/memory/DB connections), access logs, and optionally exposing client-side Prometheus metrics from workers. Details and knobs (Prefect 3.x) What timeouts/retries are used - Client request timeout: default 60s - Setting:

request_timeout

in API settings - Set via environment or profile (e.g.,

PREFECT_API_REQUEST_TIMEOUT=120

) - Ref: API health check (endpoint you can ping) - Client retries: default 5 attempts with jitter/backoff - Settings in Client settings:

max_retries

(default 5),

retry_jitter_factor

(default 0.2), and optional

retry_extra_codes

- Set via env/profile (e.g.,

PREFECT_CLIENT_MAX_RETRIES=8

,

PREFECT_CLIENT_RETRY_JITTER_FACTOR=0.3

) Observable symptoms when the API is overloaded - Worker/flow logs show

httpx.ConnectTimeout

or

httpx.ReadTimeout

when calling the API - Repeated retry logs, then failures like “failed to set flow/task run state” - Flow run remains in Running in the UI while code finished, or is eventually marked Late/Crashing - Increased lateness for scheduled runs; workers dropping heartbeats - 503/429/5xx from the server under load (the API includes a RequestLimitMiddleware to shed load) How to measure “load” on the Prefect Server API - Probe the health endpoint and track latency/error codes - Endpoint:

GET /api/health

- Example quick check

curl -s -o /dev/null -w "connect=%{time_connect} total=%{time_total} code=%{http_code}\n" http://<server>/api/health

- Load test smoke:

hey -n 1000 -c 50 http://<server>/api/health

- Doc: Health check - Server logs and access logs - Look for spikes in request time, 5xx/429/503, or messages indicating the request limiter rejected requests - Infrastructure metrics - DB: connection count, query latency, locks, CPU/IO - API container/pod: CPU, memory, restarts, network, file descriptors - Any reverse proxy/LB in front of the API: request rate, p50/p95/p99 latency, error rates - Client-side (worker/flow) metrics endpoint - Prefect can expose a tiny Prometheus exporter from the client side - Enable with:

export PREFECT_CLIENT_METRICS_ENABLED=true
    export PREFECT_CLIENT_METRICS_PORT=4201

- Then scrape the process’s metrics port (useful for seeing request/retry behavior from the client point of view) - UI indicators - Watch average flow lateness and spikes in run failures as indirect signals of API or DB stress - Self-hosting guidance - See scaling and deployment notes: Self-hosted Prefect 3 Practical mitigations if you’re hitting timeouts - Increase client-side timeout/retries for the API

export PREFECT_API_REQUEST_TIMEOUT=120
  export PREFECT_CLIENT_MAX_RETRIES=8
  export PREFECT_CLIENT_RETRY_JITTER_FACTOR=0.3

- Reduce API pressure from logs - Batch logs more and/or reduce size

export PREFECT_LOGGING_TO_API_BATCH_INTERVAL=5.0
  export PREFECT_LOGGING_TO_API_BATCH_SIZE=4000000
  export PREFECT_LOGGING_TO_API_MAX_LOG_SIZE=500000

- Scale the server horizontally and tune Postgres - Add API replicas behind a load balancer, ensure DB max_connections and CPU/IO are sufficient - Front the API with a proxy (NGINX/Envoy) to get clear metrics on RPS, latency, and errors Follow-ups - Are you self-hosting Prefect Server (version, deployment method: Docker/K8s/VM)?

- Do you have example timeout errors from your worker/flow logs? - Roughly how many concurrent runs and workers are you running when you see the issue? Helpful docs - Self-hosted Prefect 3: Self-hosted guide - Worker healthchecks: Worker health checks - API health endpoint: /api/health