<@ULVA73B9P> getting sqlalchemy.exc.TimeoutError: ...
# ask-marvin
a
@Marvin getting sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00, this in prefect server side
m
thought for 393.7 seconds
It sounds like your Prefect Server API is exhausting its database connection pool. In Prefect 3.x, the server uses SQLAlchemy’s QueuePool with these defaults: - pool_size: 5 - max_overflow: 10 - pool_timeout: 30.0s - pool_recycle: 3600s When pool_size + max_overflow are all checked out and not returned within pool_timeout, you’ll see the error you posted. Quick mitigations - Increase pool capacity (most common fix)
Copy code
# set on the Prefect Server container/process and restart the server
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=15
export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=60
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
- Confirm Postgres max_connections supports the total across all API replicas If you run N server pods, effective connections ≈ N * (pool_size + max_overflow). Ensure you’re not hitting Postgres connection limits. - Check for long transactions or leaked sessions Run this on Postgres to see usage and long-lived sessions:
Copy code
SELECT 
  count(*) AS total,
  count(*) FILTER (WHERE state = 'active') AS active,
  count(*) FILTER (WHERE state = 'idle') AS idle,
  (SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max_connections
FROM pg_stat_activity;

SELECT pid, usename, application_name, state, backend_start, state_change, wait_event_type, wait_event, query
FROM pg_stat_activity
ORDER BY state_change DESC
LIMIT 50;
- Consider PgBouncer for larger deployments For multiple API replicas or spiky traffic, front Postgres with PgBouncer so the DB handles fewer physical connections. Keep your Prefect pool moderate (e.g., pool_size 5–10) per API instance. See Database maintenance. Exact setting names and docs - pool_size - Env: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE - Default: 5 - max_overflow - Env: PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW - Default: 10 - pool_timeout - Env: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT - Default: 30.0 - pool_recycle - Env: PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE - Default: 3600 References: - Settings reference: Prefect 3 settings reference - Database settings API: Server database settings - Database maintenance: DB maintenance and monitoring A few clarifying questions to help right-size the pool: - How many Prefect Server API instances are you running (pods/containers)? - Which database are you using (Postgres version/hosted service) and what is max_connections? - Roughly how many concurrent API calls or load spikes are you seeing? - Are you using PgBouncer already? Share those and I can suggest concrete numbers for pool_size/max_overflow tailored to your setup.