Thanks for the detailed context — this pattern points to your database being hammered by worker/queue polling and background services until Postgres degrades (lock contention + cached state growth), then everything tips into “late” because the API can’t respond fast enough. That specific query is cheap by itself, but it’s executed very frequently by every worker and service. When many workers poll the same queue and the API updates

work_queue.last_polled

, the

work_queue

row becomes “hot” and small hiccups cascade into high latency. Permanent fixes that usually resolve this: A) Reduce polling pressure and hot-row contention - Use fewer, larger workers per queue; avoid many workers polling the same queue - Prefer 1 worker per queue (or a small pool) with higher

--limit

instead of many small workers - Increase the worker poll interval and prefetch window to keep throughput but slash DB QPS - In Prefect 3.x defaults are roughly 10s; set: - Workers

PREFECT_WORKER_QUERY_SECONDS=30
    PREFECT_WORKER_PREFETCH_SECONDS=60

- This trades a few seconds of pickup latency for much lower DB load B) Right-size/refresh the API’s DB connection pool - Recycle connections more aggressively (prevents long-lived connection state/caches from growing and helps Cloud SQL) - Bump pool size a bit if you have multiple API pods and steady traffic - On your API pods:

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=600
    PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
    PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
    PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=30

- If you still see plan/cache pathologies over time in Cloud SQL, you can cap/disable statement caches:

PREFECT_SERVER_DATABASE_STATEMENT_CACHE_SIZE=0
  PREFECT_SERVER_DATABASE_PREPARED_STATEMENT_CACHE_SIZE=0

Note: this trades memory stability for a bit more CPU; only apply if you see prepared-statement/generic-plan issues in Cloud SQL C) Spread load across queues where possible - Don’t point a large worker fleet at a single queue - If you need many workers (e.g., different node pools), shard deployments across multiple queues so each queue has only a few pollers D) Tune background services only if needed - If your workload is very spiky, slightly increasing service loop intervals can help reduce DB churn: - Scheduler and late-runs loops can be nudged up a bit (e.g., +5–10s) to reduce steady-state pressure without affecting behavior materially E) Postgres/Cloud SQL hygiene - Ensure autovacuum is keeping up on hot tables (work_queue, flow_run, task_run, logs) - Watch bloat and vacuum lag; run VACUUM ANALYZE or pg_repack during low traffic if needed - Verify useful indexes exist; Prefect migrations add the important ones, but confirm you have: - Primary keys on ids (you do) - Flow run scheduling paths (e.g., by work_queue_id and next/scheduled time) - Monitor and mitigate long-running transactions and lock waiters - Check

pg_stat_activity

and

pg_locks

when latency creeps up - If you observe expensive generic plans on frequently executed queries, Cloud SQL can benefit from forcing custom plans: -

plan_cache_mode = force_custom_plan

(instance-level change; test first) Quick Kubernetes env examples - Workers (Deployment/StatefulSet env):

- name: PREFECT_WORKER_QUERY_SECONDS
    value: "30"
  - name: PREFECT_WORKER_PREFETCH_SECONDS
    value: "60"

- API server pods:

- name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE
    value: "600"
  - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
    value: "10"
  - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
    value: "20"
  - name: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
    value: "30"
  # Only if you confirm statement cache issues:
  - name: PREFECT_SERVER_DATABASE_STATEMENT_CACHE_SIZE
    value: "0"
  - name: PREFECT_SERVER_DATABASE_PREPARED_STATEMENT_CACHE_SIZE
    value: "0"

Why this works

- Fewer pollers per queue + longer poll interval = far fewer hits to the

work_queue

table and far fewer

last_polled

writes (removes hot-row contention). - Prefetch widens the pickup window so runs still start on time with less polling. - Connection recycling avoids memory/cache buildup in long-lived connections (a known source of “everything is fine for days, then it falls over” in Cloud SQL). - Proper pool sizing prevents thundering herds of new connections when latency spikes. If you want, we can help tune to your footprint. A few quick questions: - How many workers do you run per queue? How many total queues? - What are your current

PREFECT_WORKER_QUERY_SECONDS

and

PREFECT_WORKER_PREFETCH_SECONDS

? - How many API pods are you running, and what are their current DB pool settings? - Typical scheduled runs per minute? Any spikes? - Cloud SQL instance size and autovacuum settings? Background reading if helpful: - More memory, more problems: lessons from scaling Postgres-backed services on Cloud SQL Once we have the numbers above, I can suggest concrete values for your environment and a rollout plan to make this stick.