<@ULVA73B9P> - i am using self hosted prefect - 3....
# ask-marvin
c
@Marvin - i am using self hosted prefect - 3.6.7 . My requirement is to use multiple hosts of PostgreSQL in connection string. Ex: postgresql+asyncpg://user@/dbname?host=host15432&amp;host=host25432&gsslib=gssapi&krbsrvname=postgresql&ssl=require&target_session_attrs=primary With above format, i see prefect is bit slow. Any recommendation to use multi host, as part of Prefect DB Connection String.
m
thought for 274.8 seconds
Short answer: In Prefect 3.6.7 (self‑hosted), the API server uses SQLAlchemy with the asyncpg driver. Multi‑host/failover libpq-style connection strings (e.g., multiple host params,
target_session_attrs
,
gsslib
,
krbsrvname
) are not supported by asyncpg and can introduce connection delays as it probes/ignores unsupported options. The recommended approach is to point Prefect at a single, highly‑available endpoint (managed writer endpoint, HAProxy/pgpool/pgBouncer, etc.) rather than listing multiple hosts in the URL. What to do instead - Prefer a single logical endpoint that handles failover: - Managed DB writer endpoint (e.g., RDS Multi‑AZ writer), a VIP/Load Balancer, or pgBouncer/pgpool in front of your replicas. - Then configure Prefect with a single URL using asyncpg:
Copy code
export PREFECT_SERVER_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:password@db-endpoint:5432/prefect>"
- If you must handle host failover client‑side: - Use DNS that returns multiple A records for one hostname and point Prefect to that single hostname. - Reduce connection establishment wait so a down host doesn’t stall the API:
Copy code
export PREFECT_SERVER_DATABASE_CONNECTION_TIMEOUT=3
- Right‑size the pool so Prefect spends less time creating new connections under load:
Copy code
export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=20
    export PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=10
    export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_TIMEOUT=30
    export PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_RECYCLE=1800
- If using pgBouncer (transaction mode), disable statement caching to avoid errors and extra latency:
Copy code
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_STATEMENT_CACHE_SIZE=0
- TLS/mTLS: prefer Prefect’s connect args over
ssl=require
(libpq option). For asyncpg use:
Copy code
export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_ENABLED=true
  # Optional:
  export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CA_FILE=/path/ca.crt
  export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_CERT_FILE=/path/client.crt
  export PREFECT_SERVER_DATABASE_SQLALCHEMY_CONNECT_ARGS_TLS_KEY_FILE=/path/client.key
- Kerberos/GSSAPI: asyncpg does not support GSSAPI auth; Prefect 3.x does not support psycopg(3) as its async driver. If you require Kerberos, you’ll need to place a proxy in front of Postgres that terminates Kerberos and exposes standard user/password or IAM auth to Prefect. Why your current URL feels slow - The parameters you’re using (
host=host1&host=host2
,
gsslib
,
krbsrvname
,
target_session_attrs
) are libpq/psycopg features. With asyncpg they are not supported and can cause: - Sequential connection attempts to multiple hosts until success/timeout - Extra round‑trips while the driver ignores unknown options - Longer API startup and connection pool warm‑up times Docs and references - Self‑hosted guide (HA patterns, Postgres/Redis guidance): Self-hosted Prefect (v3) - Settings reference (all DB settings, connect args, TLS): Settings reference - Maintenance guidance (pooling, performance): Database maintenance If you can share a bit more: - Are you using Kerberos/GSS to auth to Postgres? - Is the “slow” behavior during API startup, or ongoing (e.g., latency on API requests/flow runs)? - Are you running through pgBouncer/pgpool already? With those details I can suggest tighter timeouts/pool sizing (or a proxy pattern) tailored to your setup.
c
@Marvin - can prefect use psycopg2 driver instead of asyncpg for DB connection
m
thought for 22.6 seconds
Short answer: No. In Prefect 3.6.7 the API/Server requires an async SQLAlchemy engine. For Postgres that means
postgresql+asyncpg
. Synchronous drivers like
psycopg2
aren’t supported, and the psycopg(3) async dialect isn’t supported for the server either. Use:
Copy code
export PREFECT_SERVER_DATABASE_CONNECTION_URL="<postgresql+asyncpg://user:password@host:5432/prefect>"
If you need features typically tied to psycopg/Libpq (e.g., Kerberos/GSS), put a proxy in front of Postgres (pgBouncer/pgpool/load balancer or a managed writer endpoint) and keep Prefect pointed at that single endpoint. You can still use psycopg in your flow code for your own databases; this limitation applies only to the Prefect orchestration database. Docs: - Self-hosted guide: Self-hosted Prefect (v3) - Settings reference (DB and connect args): Settings reference