Thanks for flagging this — 128 concurrent, short-lived flows can put a lot of write pressure on a self-hosted API/DB, which the UI depends on. A few questions to pinpoint the bottleneck: - Prefect version (assuming 3.x unless you say otherwise)? - DB engine: SQLite or Postgres? Any PgBouncer? - Are flow/task logs being sent to the API (default is on)? - Rough hardware for the API/DB? - Any reverse proxy/load balancer in front of the API? Quick wins (in order of impact) - Move off SQLite if you’re using it. Use Postgres for 100+ concurrent runs. - Reduce logging-to-API load: - Either turn it off:

PREFECT_LOGGING_TO_API_ENABLED=false

- Or reduce frequency and increase batching:

PREFECT_LOGGING_TO_API_BATCH_INTERVAL=10.0
    PREFECT_LOGGING_TO_API_BATCH_SIZE=8000000

- Shorten events retention (the events table can get huge fast):

PREFECT_EVENTS_RETENTION_PERIOD=2d

- Increase DB connection pooling (per API process):

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=25
  PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=40
  PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=30.0
  PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=3600

If you put Postgres behind PgBouncer in transaction mode, also set:

PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0

- Worker heartbeat/query tuning (reduces DB chatter):

PREFECT_WORKER_HEARTBEAT_SECONDS=60
  PREFECT_WORKER_QUERY_SECONDS=15
  PREFECT_WORKER_PREFETCH_SECONDS=15

- API keep-alives if you’re behind a LB:

PREFECT_SERVER_API_KEEPALIVE_TIMEOUT=65

Or start with:

prefect server start --keep-alive-timeout 65

Scale-out pattern (if one API process isn’t enough) - Run multiple API server instances behind a load balancer, and run one instance of services separately. Example: - Several API-only instances:

prefect server start --no-services

- One services instance:

prefect server start --no-ui

- Use Postgres (ideally via PgBouncer) and Redis for server messaging if you scale horizontally. - Connection URL example:

PREFECT_API_DATABASE_CONNECTION_URL=<postgresql+asyncpg://prefect:***@pgbouncer:6432/prefect>

Why this helps - Logging-to-API and event writes are the top sources of DB write load during high concurrency; reducing log frequency and event retention keeps the DB responsive. - The default SQLAlchemy pool (pool_size=5, max_overflow=10) is too small for 100+ concurrent runs; increasing it prevents request queueing/timeouts. - Longer worker heartbeat intervals reduce steady-state write traffic. - If you’re on SQLite, concurrent writes will cause lock contention and UI/API stalls; Postgres is required at this scale. If you can share: - Whether you’re on SQLite vs Postgres - Current env settings (

prefect config view

output) - A quick sample of server logs around the slow periods (timeouts/pool exhaustion messages) …I can tailor exact values for your setup and provide a minimal config you can drop in.

Got it — here are the Prefect 3.x defaults for the settings we discussed: Logging to API - PREFECT_LOGGING_TO_API_ENABLED: true - PREFECT_LOGGING_TO_API_BATCH_INTERVAL: 2.0 - PREFECT_LOGGING_TO_API_BATCH_SIZE: 4000000 - PREFECT_LOGGING_TO_API_MAX_LOG_SIZE: 1000000 Events retention - PREFECT_EVENTS_RETENTION_PERIOD: 7d Database (SQLAlchemy) pooling and timeouts - PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE: 5 - PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW: 10 - PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT: 30.0 - PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE: 3600 - PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE: unset (driver default) - PREFECT_SERVER_DATABASE_TIMEOUT: 10.0 - PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT: 5.0 API server - PREFECT_SERVER_API_KEEPALIVE_TIMEOUT: 5 - PREFECT_SERVER_API_DEFAULT_LIMIT: 200 Worker polling/heartbeat - PREFECT_WORKER_HEARTBEAT_SECONDS: 30 - PREFECT_WORKER_QUERY_SECONDS: 10 - PREFECT_WORKER_PREFETCH_SECONDS: 10 References - Settings reference: Prefect 3 settings - CLI help for server keepalive: run

prefect server start --help

If you want defaults for any other specific setting, tell me which ones and I’ll add them.

Great question. Short answer: no — they’re not all “server-level”. Here’s how to scope them: Rule of thumb - PREFECT_SERVER_* => set on Prefect Server processes (API and/or services) - PREFECT_WORKER_* => set on Prefect worker processes (the things polling work pools) - PREFECT_LOGGING_* => set on any Prefect process that emits logs (flows, workers, server services) - PgBouncer settings => set in PgBouncer’s own INI config, not in Prefect Mapping the ones we discussed - Logging to API (PREFECT_LOGGING_TO_API_*): set on your flow/worker runtime containers (and optionally on server services if you want to cut their API logs too). This controls client-side batching to the API; the server can’t change batching on behalf of clients. - Events retention (PREFECT_EVENTS_RETENTION_PERIOD): server-side. Set on the Prefect Server services process (or the “all-in-one” server if you run API+services together). - DB pool/timeouts (PREFECT_SERVER_DATABASE_SQLALCHEMY_* and *_TIMEOUT): server-side. Apply to all Prefect Server processes that talk to the DB (API and services). Workers do not use these. - API keepalive/default limit (PREFECT_SERVER_API_KEEPALIVE_TIMEOUT, PREFECT_SERVER_API_DEFAULT_LIMIT): server-side. Apply to API processes. - Worker heartbeat/polling (PREFECT_WORKER_HEARTBEAT_SECONDS, PREFECT_WORKER_QUERY_SECONDS, PREFECT_WORKER_PREFETCH_SECONDS): worker-side. Apply to worker processes only. - Statement cache size for PgBouncer (PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0): server-side (API/services) when you put Postgres behind PgBouncer in transaction mode. - PgBouncer pool sizing (pool_mode, default_pool_size, etc.): configure in PgBouncer’s INI; not Prefect. Notes - If you run “all-in-one” server (prefect server start) then setting PREFECT_SERVER_* in that container covers both API and services. If you split API-only and a separate services instance, set the server vars on both. - To reduce DB load from logs, prioritize setting PREFECT_LOGGING_TO_API_* on your flow/worker images. Docs - Settings reference: Prefect 3 settings If you share how you’re deploying (single container, Docker Compose, Helm) I can point to exactly where to put each var.