<@ULVA73B9P> can you explain this error, :arrow_up...
# marvin-ai
k
@Marvin can you explain this error, ⬆️ Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: asyncio.exceptions.TimeoutError ← this is the main error (connection timed out) Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: return current.parent.switch(awaitable) # type: ignore[no-any-return,attr-defined] # noqa: E501 Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib64/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: value = await result Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib64/python3.9/site-packages/asyncpg/connection.py", line 2421, in connect Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: return await connect_utils._connect( Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib64/python3.9/site-packages/async_timeout/__init__.py", line 179, in aexit Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: self._do_exit(exc_type) Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: File "/home/ec2-user/prefect-dbt/prefect-dbt-env/lib64/python3.9/site-packages/async_timeout/__init__.py", line 265, in _do_exit Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: raise asyncio.TimeoutError Oct 06 094728 ip-XXX-XX-XX-XXX.ec2.internal prefect[5534]: asyncio.exceptions.TimeoutError my prefect is using the postgres db
m
thought for 133.2 seconds
Short version: that stack trace is asyncpg timing out while Prefect (via SQLAlchemy) tries to open a TCP connection to your Postgres database. It’s almost always a networking/connectivity issue (wrong host/port, security group/firewall/VPC routing, DB not listening) or, less commonly, an aggressive connection timeout. What to check first - Can you reach the DB from the same EC2 host? - Try:
psql "host=<host> port=<port> dbname=<db> user=<user> sslmode=require" -c 'select 1'
- Or TCP check:
nc -vz <host> <port>
(should say “succeeded”) - Security groups/NACLs/routes: - Inbound on the DB (e.g., RDS) allows 5432 from your EC2 instance’s SG. - If the DB is on another host, Postgres is listening on 0.0.0.0 (not just localhost). - Connection URL is correct and uses the async driver: - It should look like:
<postgresql+asyncpg://USER:PASS@HOST:5432/DBNAME>
- If your provider requires TLS, include it, e.g.:
...?sslmode=require
- The DB is up and healthy (logs show it accepting connections). Optional: increase Prefect’s DB timeouts/pool limits In Prefect 3.x you can tune these with environment variables (server-side): - Increase connection establishment timeout (default ~5s): -
PREFECT_SERVER_DATABASE__CONNECTION_TIMEOUT=15
- Increase general DB operation timeout (default ~10s): -
PREFECT_SERVER_DATABASE__TIMEOUT=30
- Adjust pool if you’re under load (to avoid pool checkout timeouts): -
PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_SIZE=10
-
PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_TIMEOUT=60
-
PREFECT_SERVER_DATABASE__SQLALCHEMY__MAX_OVERFLOW=20
Example env setup
Copy code
export PREFECT_SERVER_DATABASE__CONNECTION_URL='<postgresql+asyncpg://user:pass@your-host:5432/prefect?sslmode=require>'
export PREFECT_SERVER_DATABASE__CONNECTION_TIMEOUT=15
export PREFECT_SERVER_DATABASE__TIMEOUT=30
export PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_SIZE=10
export PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_TIMEOUT=60
export PREFECT_SERVER_DATABASE__SQLALCHEMY__MAX_OVERFLOW=20
References - Prefect 3 database configuration (AsyncPostgresConfiguration) - Prefect 3 settings reference A couple of clarifying questions to zero in: - Where exactly are you seeing this error: Prefect Server (API) logs on the EC2 host, or inside a flow/worker that connects to your own application database? - What’s the DB host (RDS? another EC2?) and can you
psql
to it from the same EC2 machine? - Could you share your connection URL format (redact user/pass), and whether TLS is required by your DB?