Chalama Reddy
02/18/2026, 2:10 AMMarvin
02/18/2026, 2:10 AMMarvin
02/18/2026, 2:13 AMpostgresql+asyncpg://...), the Prefect database URL cannot include a host list either.
Recommended approaches for HA/failover:
- Use a single HA endpoint in your DSN:
- Managed Postgres (e.g., AWS RDS/Aurora): use the cluster/writer endpoint that fails over automatically
- A proxy/load balancer (PgBouncer, HAProxy, pgpool-II), or a virtual IP that tracks the current primary
- DNS-based failover (CNAME that points at the current primary)
- Point Prefect at that single stable endpoint; the proxy/managed service handles failover.
Example (Prefect 3.x):
export PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:password@db-ha-endpoint.example.com:5432/prefect>"
If you share which HA setup you’re using (e.g., Patroni, RDS, self-managed with HAProxy/PgBouncer), I can suggest a concrete connection pattern for that environment.Chalama Reddy
02/18/2026, 2:17 AMChalama Reddy
02/18/2026, 2:17 AMMarvin
02/18/2026, 2:18 AMMarvin
02/18/2026, 2:22 AMpostgresql+asyncpg://... and pass multiple hosts using SQLAlchemy’s multi-host syntax (query parameters). Example:
export PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:pass@localhost/prefect?host=pg-primary.example.com,pg-replica.example.com&port=5432,5432>"
Notes:
- Use the query parameters for host and port with matching counts.
- You can also repeat the host parameter instead of comma-separating:
<postgresql+asyncpg://user:pass@localhost/prefect?host=pg-primary.example.com:5432&host=pg-replica.example.com:5432>
- Keep the +asyncpg driver in the URL.
- Failover occurs on connection attempts; existing connections won’t move mid-transaction.
Docs: SQLAlchemy: Specifying Multiple Fallback Hosts
If you’d like, share your HA topology (e.g., Patroni, RDS/Aurora, pgpool/PgBouncer) and I can suggest a concrete DSN pattern and any additional considerations.Chalama Reddy
02/18/2026, 2:27 AMMarvin
02/18/2026, 2:27 AMMarvin
02/18/2026, 2:30 AMpool_size=5, max_overflow=10), bursts of API/UI traffic can trigger new connections more often, amplifying any per-connect overhead from multi-host.
- Timeouts not tuned: If a host is down and connect timeout is high, each attempt waits that long before moving on.
- Mixed primary/replica in the list: If a replica accepts connections but is read-only, you’ll see errors on writes or retries that look like “slowness.”
- DNS and TLS handshakes across multiple endpoints can add latency on each new connect.
Quick mitigations to try
1) Prefer a single stable HA endpoint
- If you have a managed service (e.g., RDS/Aurora), use the cluster writer endpoint.
- Or front Postgres with PgBouncer/HAProxy and point Prefect at that single endpoint.
- This eliminates sequential fallback on each new connection.
2) Tune connection and pool settings
- Increase pool size so you make fewer new connections:
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
- Keep connect timeouts low so failover happens fast if the first host is down:
PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=5
- Make sure the healthy/writer host is first in your URL’s host list.
3) If using PgBouncer (recommended in PROD)
- Use one DSN to PgBouncer and let it manage failover/pooling.
- If PgBouncer is in transaction pooling mode, disable prepared statement caching to avoid cross-connection prepare issues:
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE=0
- Optionally set:
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME=prefect-server
4) Validate multi-host URL ordering and health
- Remove any hosts that are intermittently reachable or firewalled.
- Place the primary/writer first.
- Reference (multi-host syntax): SQLAlchemy: Specifying Multiple Fallback Hosts
How to confirm it’s the DB connection path
- Temporarily switch the DSN to a single writer endpoint and compare UI latency.
- Watch Postgres logs for repeated connection attempts and timeouts.
- Look for API logs showing connection checkout waits close to PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT.
If you can share:
- Your exact DSN (redact creds), which HA stack you’re on (Patroni, RDS/Aurora, etc.), and whether any host in the list is intermittently down
- Current pool/timeouts you’ve set
- Approximate DB size (number of flow runs, task runs)
…I can suggest a concrete, minimal-change configuration to fix the slowness.Chalama Reddy
02/18/2026, 2:45 AMMarvin
02/18/2026, 2:45 AMMarvin
02/18/2026, 2:46 AM<postgresql+asyncpg://user@/dbname?host=HostA:5432&host=HostB:5432&host=HostC:5432&gsslib=gssapi&krbsrvname=postgresql&ssl=require&target_session_attrs=primary>
This is a valid SQLAlchemy multi-host URL for asyncpg and should work. That said, the UI slowness you’re seeing in PROD when multi-host is enabled is a common symptom of connection behavior rather than a Prefect 3.3.5 bug.
Why multi-host can slow things down
- Sequential fallback on connect: each new connection will try HostA, then HostB, then HostC (and with target_session_attrs=primary, even a reachable replica will be rejected before trying the next host). Any slow-to-fail host in the list adds latency per new connection.
- Kerberos + SSL handshakes: GSSAPI and TLS add noticeable overhead for each fresh connection.
- Small pool sizes: with defaults, bursts of API/UI traffic trigger more new connects, amplifying the above.
- Timeouts: if a host is down and connection timeout is high, each attempt waits that long before moving on.
Actionable mitigations
1) Put the writer first (or remove unhealthy hosts)
- Ensure the current primary is first in your host list. With target_session_attrs=primary, placing a replica first incurs an extra round-trip for every new connection.
- Remove any intermittently failing host from the list.
2) Tune pool and timeouts to minimize new connections and fail fast
- Increase pool size so you reuse connections more and handshake less:
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
- Keep checkout waits bounded and recycle periodically:
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
- Lower connect timeout so fallback proceeds quickly if a host is down or not primary:
PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=5
- Use the stable DB URL env var:
PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user@/dbname?host=HostA:5432&host=HostB:5432&host=HostC:5432&gsslib=gssapi&krbsrvname=postgresql&ssl=require&target_session_attrs=primary>"
3) Consider a single stable HA endpoint
- If possible, use a managed writer/cluster endpoint or place PgBouncer/HAProxy in front and point Prefect at that single endpoint. This removes sequential fallback from the app path and tends to eliminate the UI latency spikes.
How to verify it’s the connection path
- Temporarily point Prefect at only the current primary and compare UI latency.
- Watch Postgres logs for frequent connection attempts and timeouts during UI loads.
- Check API logs for long connection checkout waits (near your PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT).
Reference for the multi-host syntax you’re using: SQLAlchemy: Specifying Multiple Fallback Hosts
If you can share:
- Which host in your list is the writer, and whether any are intermittently down
- Current values for pool/timeouts
- Rough traffic/throughput during the slow periods
…I can recommend a minimal-change config (host order and env vars) tailored to your setup.