Thanks for the details. In self‑hosted Prefect 3.6.7, the “TooManyConnections” errors almost always come from the API’s SQLAlchemy pool exhausting Postgres connections across one or more API processes. Quick context (Prefect 3.x): - Only the Prefect API connects to Postgres. Workers/flows talk to the API over HTTP and do not connect to Postgres directly. - Each API process has its own SQLAlchemy connection pool. By default: - pool_size=5 - max_overflow=10 - So each API process can use up to 15 DB connections. - If you run multiple API processes (e.g., multiple uvicorn workers via

--workers

or multiple replicas/pods), total possible connections = processes × (pool_size + max_overflow). Why you’re likely seeing it with a 100-connection cap - With the defaults, 7 API processes can consume up to 7 × 15 = 105 connections, not counting other apps or admin connections. - Background API tasks (cleanup, scheduling, leases, etc.) also borrow connections from the same pool and can hold them under load. What to check first 1) How many API processes are you running? - If you run

prefect server start --workers N

, total connections scale linearly with N. In 3.6.7 there is a

--workers

flag; default is 1. - If you’re in Kubernetes or Compose, check if you’ve scaled the API deployment to multiple replicas. 2) Whether you’ve already tuned the pool via env vars (see below). 3) Whether you’re running PgBouncer in front of Postgres. How to cap/shape the pool to fit under 100 - Start by clamping max overflow and increasing timeout, so bursts queue instead of failing:

# Cap each API process at 5 connections and wait longer before erroring
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=5
export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=0
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=3600

- If you have multiple API processes/replicas, multiply pool_size across them and ensure the sum stays comfortably below 100 (account for other DB clients too). For example, with 4 API processes and a 100-connection DB cap, a safe starting point is pool_size=5, max_overflow=0 → 4 × 5 = 20 total. - If you need higher concurrency from the API layer, either: - Increase Postgres max_connections above 100, or - Put PgBouncer in front of Postgres and point Prefect at PgBouncer (transaction pooling). If you do this, disable asyncpg’s statement cache:

# Point Prefect at PgBouncer and disable statement cache for transaction pooling
export PREFECT_SERVER_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:pass@pgbouncer:6432/prefect>"
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0

Notes on workers -

prefect server start --workers N

controls uvicorn processes for the API/UI. Each process has its own pool, so N increases DB connections linearly. There’s no env var equivalent for

--workers

. - In multi-worker mode (

--workers > 1

), services are not run in-process and Redis is required. If you don’t need multiple API processes, stick with 1–2. How to verify/tune - See your effective settings:

prefect config view | grep -i server.database

- Monitor connections in Postgres:

SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;
SELECT application_name, count(*) FROM pg_stat_activity WHERE datname='prefect' GROUP BY application_name;

- For deeper debugging:

export PREFECT_SERVER_DATABASE_ECHO=1
export PREFECT_SERVER_LOGGING_LEVEL=DEBUG

Source references - Database configuration (pool settings and connect args): Database configuration source - Server CLI (workers flag): Server CLI source

- Background tasks that can hold connections: Background workers If you can share: - How you’re running the API (single process vs

--workers N

, or number of replicas), - Whether PgBouncer is in use, - Any pool env vars you’ve already set, I can help you pick concrete pool values that keep you safely under your 100-connection limit.

@Marvin - We are using 21 Workers with 4 API servers

Got it — thanks. With 21 Prefect workers and 4 API servers, the DB pressure is almost certainly coming from the API layer. Key points: - Prefect workers do not connect to Postgres. They call the API over HTTP. - Each API process has its own SQLAlchemy pool. By default, per process: pool_size=5 and max_overflow=10 → up to 15 DB connections per process. - Total DB connections ≈ API replicas × uvicorn workers per replica × (pool_size + max_overflow). To keep you under a 100-connection cap, you’ll want to: 1) Confirm how many uvicorn workers each API server runs (e.g., via

prefect server start --workers N

or your container’s command/args). If each of your 4 API servers uses, say, 4 uvicorn workers, that’s 16 processes total, and with defaults you could hit 16 × 15 = 240 connections. 2) Clamp the SQLAlchemy pool per process so total stays safely below 100. A good, safe starting point: - If you have 4 API servers and 1 uvicorn worker each: - Set pool_size=8, max_overflow=0 → 4 × 8 = 32 total DB connections - If you have 4 API servers and 4 uvicorn workers each (16 processes): - Set pool_size=5, max_overflow=0 → 16 × 5 = 80 total DB connections Recommended env vars (apply to each API server/pod):

# Cap connections per API process
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=5
export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=0

# Make bursts wait instead of failing fast
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60

# Reasonable recycle to avoid stale connections
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=3600

# Optional: make connections easy to identify in pg_stat_activity
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME=prefect-api

If you’re using PgBouncer, point the API to PgBouncer and disable asyncpg’s statement cache for transaction pooling:

export PREFECT_SERVER_DATABASE_CONNECTION_URL="<postgresql+asyncpg://USER:PASS@pgbouncer:6432/prefect>"
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0

Also verify/adjust the number of uvicorn workers: - CLI:

prefect server start --workers N

(default is 1) - If you are running multiple API replicas, multiply by replicas Source: Server CLI source Where these pool settings come from: - Database configuration (SQLAlchemy pool) How to confirm what’s active:

# See effective settings on a running API server
prefect config view | grep -i server.database

# In Postgres
SELECT application_name, count(*) 
FROM pg_stat_activity 
WHERE datname = 'prefect'
GROUP BY application_name;

A couple of quick clarifying questions so I can give you exact numbers: - How many uvicorn workers are running per API server (per replica/pod)? - Are you using PgBouncer? - Have you already set any of the

PREFECT_SERVER_DATABASE_SQLALCHEMY_*

env vars? Once I have that, I’ll calculate precise pool_size/max_overflow values that keep you well under the 100-connection limit with headroom.