<@ULVA73B9P> - Hi. i am using self hosted prefect ...
# ask-marvin
c
@Marvin - Hi. i am using self hosted prefect and we have a PostgreSQL with HA. My requirement is to use multiple hosts of PostgreSQL in connection string. Does asyncpg supports it?
m
thought for 171.3 seconds
Short answer: no — asyncpg does not support specifying multiple PostgreSQL hosts in a single connection string. Because Prefect’s self-hosted API uses SQLAlchemy with the asyncpg driver (in Prefect 3.x the DB URL must be
postgresql+asyncpg://...
), the Prefect database URL cannot include a host list either. Recommended approaches for HA/failover: - Use a single HA endpoint in your DSN: - Managed Postgres (e.g., AWS RDS/Aurora): use the cluster/writer endpoint that fails over automatically - A proxy/load balancer (PgBouncer, HAProxy, pgpool-II), or a virtual IP that tracks the current primary - DNS-based failover (CNAME that points at the current primary) - Point Prefect at that single stable endpoint; the proxy/managed service handles failover. Example (Prefect 3.x):
Copy code
export PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:password@db-ha-endpoint.example.com:5432/prefect>"
If you share which HA setup you’re using (e.g., Patroni, RDS, self-managed with HAProxy/PgBouncer), I can suggest a concrete connection pattern for that environment.
c
As per SQLAlchemy - https://docs.sqlalchemy.org/en/20/dialects/postgresql.html#specifying-multiple-fallback-hosts. It supports the multiple hosts. i am using prefect-3.3.5
@Marvin - As per SQLAlchemy - https://docs.sqlalchemy.org/en/20/dialects/postgresql.html#specifying-multiple-fallback-hosts. It supports the multiple hosts. i am using prefect-3.3.5
m
thought for 260.9 seconds
You’re right — SQLAlchemy supports multiple fallback hosts, and you can use that with Prefect 3.3.5’s asyncpg dialect. In Prefect 3.x, point the server at Postgres via
postgresql+asyncpg://...
and pass multiple hosts using SQLAlchemy’s multi-host syntax (query parameters). Example:
Copy code
export PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:pass@localhost/prefect?host=pg-primary.example.com,pg-replica.example.com&port=5432,5432>"
Notes: - Use the query parameters for
host
and
port
with matching counts. - You can also repeat the
host
parameter instead of comma-separating:
Copy code
<postgresql+asyncpg://user:pass@localhost/prefect?host=pg-primary.example.com:5432&host=pg-replica.example.com:5432>
- Keep the
+asyncpg
driver in the URL. - Failover occurs on connection attempts; existing connections won’t move mid-transaction. Docs: SQLAlchemy: Specifying Multiple Fallback Hosts If you’d like, share your HA topology (e.g., Patroni, RDS/Aurora, pgpool/PgBouncer) and I can suggest a concrete DSN pattern and any additional considerations.
c
@Marvin - We are experiencing a Prefect UI loading slowness in PROD, when you using multiple hosts of postgreSQL. Is that something an issue with prefect-3.3.5 version?
m
thought for 190.1 seconds
Short answer: there isn’t a known Prefect 3.3.5 bug that makes the UI slow specifically because you’re using SQLAlchemy’s multi-host Postgres URLs. When we’ve seen this in practice, the slowness comes from connection behavior with multi-host fallback (sequential connection attempts, timeouts on down/blocked hosts, small pool sizes causing more frequent new connects), not from the UI code itself. What typically causes slowness with multi-host - Sequential fallback on connect: SQLAlchemy/asyncpg will try hosts in order. If the first (or any prior) host is slow to refuse or intermittently reachable, each new connection can incur that delay. - Small connection pool: With defaults (
pool_size=5
,
max_overflow=10
), bursts of API/UI traffic can trigger new connections more often, amplifying any per-connect overhead from multi-host. - Timeouts not tuned: If a host is down and connect timeout is high, each attempt waits that long before moving on. - Mixed primary/replica in the list: If a replica accepts connections but is read-only, you’ll see errors on writes or retries that look like “slowness.” - DNS and TLS handshakes across multiple endpoints can add latency on each new connect. Quick mitigations to try 1) Prefer a single stable HA endpoint - If you have a managed service (e.g., RDS/Aurora), use the cluster writer endpoint. - Or front Postgres with PgBouncer/HAProxy and point Prefect at that single endpoint. - This eliminates sequential fallback on each new connection. 2) Tune connection and pool settings - Increase pool size so you make fewer new connections:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
  PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
  PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=10
  PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
- Keep connect timeouts low so failover happens fast if the first host is down:
Copy code
PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=5
- Make sure the healthy/writer host is first in your URL’s
host
list. 3) If using PgBouncer (recommended in PROD) - Use one DSN to PgBouncer and let it manage failover/pooling. - If PgBouncer is in transaction pooling mode, disable prepared statement caching to avoid cross-connection prepare issues:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_PREPARED_STATEMENT_CACHE_SIZE=0
- Optionally set:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_APPLICATION_NAME=prefect-server
4) Validate multi-host URL ordering and health - Remove any hosts that are intermittently reachable or firewalled. - Place the primary/writer first. - Reference (multi-host syntax): SQLAlchemy: Specifying Multiple Fallback Hosts How to confirm it’s the DB connection path - Temporarily switch the DSN to a single writer endpoint and compare UI latency. - Watch Postgres logs for repeated connection attempts and timeouts. - Look for API logs showing connection checkout waits close to
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
. If you can share: - Your exact DSN (redact creds), which HA stack you’re on (Patroni, RDS/Aurora, etc.), and whether any host in the list is intermittently down - Current pool/timeouts you’ve set - Approximate DB size (number of flow runs, task runs) …I can suggest a concrete, minimal-change configuration to fix the slowness.
c
@Marvin - Connection string used in format - "postgresql+asyncpg://user@/dbname?host=HostA5432&amp;host=HostB5432&host=HostC:5432"&gsslib=gssapi&krbsrvname=postgresql&ssl=require&target_session_attrs=primary
m
thought for 69.8 seconds
Thanks for sharing the DSN. Decoding the HTML entities, it looks like you’re using this form:
Copy code
<postgresql+asyncpg://user@/dbname?host=HostA:5432&host=HostB:5432&host=HostC:5432&gsslib=gssapi&krbsrvname=postgresql&ssl=require&target_session_attrs=primary>
This is a valid SQLAlchemy multi-host URL for asyncpg and should work. That said, the UI slowness you’re seeing in PROD when multi-host is enabled is a common symptom of connection behavior rather than a Prefect 3.3.5 bug. Why multi-host can slow things down - Sequential fallback on connect: each new connection will try HostA, then HostB, then HostC (and with
target_session_attrs=primary
, even a reachable replica will be rejected before trying the next host). Any slow-to-fail host in the list adds latency per new connection. - Kerberos + SSL handshakes: GSSAPI and TLS add noticeable overhead for each fresh connection. - Small pool sizes: with defaults, bursts of API/UI traffic trigger more new connects, amplifying the above. - Timeouts: if a host is down and connection timeout is high, each attempt waits that long before moving on. Actionable mitigations 1) Put the writer first (or remove unhealthy hosts) - Ensure the current primary is first in your
host
list. With
target_session_attrs=primary
, placing a replica first incurs an extra round-trip for every new connection. - Remove any intermittently failing host from the list. 2) Tune pool and timeouts to minimize new connections and fail fast - Increase pool size so you reuse connections more and handshake less:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
  PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
- Keep checkout waits bounded and recycle periodically:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=10
  PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
- Lower connect timeout so fallback proceeds quickly if a host is down or not primary:
Copy code
PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=5
- Use the stable DB URL env var:
Copy code
PREFECT_API_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user@/dbname?host=HostA:5432&host=HostB:5432&host=HostC:5432&gsslib=gssapi&krbsrvname=postgresql&ssl=require&target_session_attrs=primary>"
3) Consider a single stable HA endpoint - If possible, use a managed writer/cluster endpoint or place PgBouncer/HAProxy in front and point Prefect at that single endpoint. This removes sequential fallback from the app path and tends to eliminate the UI latency spikes. How to verify it’s the connection path - Temporarily point Prefect at only the current primary and compare UI latency. - Watch Postgres logs for frequent connection attempts and timeouts during UI loads. - Check API logs for long connection checkout waits (near your
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT
). Reference for the multi-host syntax you’re using: SQLAlchemy: Specifying Multiple Fallback Hosts If you can share: - Which host in your list is the writer, and whether any are intermittently down - Current values for pool/timeouts - Rough traffic/throughput during the slow periods …I can recommend a minimal-change config (host order and env vars) tailored to your setup.