hi im trying to use my postgres db running on rds as the dat

Question

hi im trying to use my postgres db running on rds as the database for orion self hosted instance backend but it continuously runs into time out errors has anyone implemented this before or facing the same issue

Deepanshu Aggarwal · Answer

```Encountered exception in request Traceback most recent call last File usr local lib python3 9 site packages starlette middleware errors py line 162 in call await scope receive send File usr local lib python3 9 site packages starlette middleware exceptions py line 75 in call raise exc File usr local lib python3 9 site packages starlette middleware exceptions py line 64 in call await scope receive sender File usr local lib python3 9 site packages fastapi middleware asyncexitstack py line 21 in call raise e File usr local lib python3 9 site packages fastapi middleware asyncexitstack py line 18 in call await scope receive send File usr local lib python3 9 site packages starlette routing py line 680 in call await route handle scope receive send File usr local lib python3 9 site packages starlette routing py line 275 in handle await scope receive send File usr local lib python3 9 site packages starlette routing py line 65 in app response = await func request File usr local lib python3 9 site packages prefect orion utilities server py line 101 in handle response scoped depends response = await default handler request File usr local lib python3 9 site packages fastapi routing py line 231 in app raw response = await run endpoint function File usr local lib python3 9 site packages fastapi routing py line 160 in run endpoint function return await dependant call values File usr local lib python3 9 site packages prefect orion api block documents py line 62 in read block documents result = await models block documents read block documents File usr local lib python3 9 site packages prefect orion database dependencies py line 117 in async wrapper return await fn args kwargs File usr local lib python3 9 site packages prefect orion models block documents py line 356 in read block documents result = await session execute File usr local lib python3 9 site packages sqlalchemy ext asyncio session py line 215 in execute result = await greenlet spawn File usr local lib python3 9 site packages sqlalchemy util concurrency py3k py line 126 in greenlet spawn result = context throw sys exc info File usr local lib python3 9 site packages sqlalchemy orm session py line 1712 in execute result = conn execute 20 statement params or execution options File usr local lib python3 9 site packages sqlalchemy engine base py line 1705 in execute 20 return meth self args 10style kwargs 10style execution options File usr local lib python3 9 site packages sqlalchemy sql elements py line 333 in execute on connection return connection execute clauseelement File usr local lib python3 9 site packages sqlalchemy engine base py line 1572 in execute clauseelement ret = self execute context File usr local lib python3 9 site packages sqlalchemy engine base py line 1943 in execute context self handle dbapi exception File usr local lib python3 9 site packages sqlalchemy engine base py line 2128 in handle dbapi exception util raise exc info 1 with traceback=exc info 2 File usr local lib python3 9 site packages sqlalchemy util compat py line 208 in raise raise exception File usr local lib python3 9 site packages sqlalchemy engine base py line 1900 in execute context self dialect do execute File usr local lib python3 9 site packages sqlalchemy engine default py line 736 in do execute cursor execute statement parameters File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 479 in execute self adapt connection await File usr local lib python3 9 site packages sqlalchemy util concurrency py3k py line 68 in await only return current driver switch awaitable File usr local lib python3 9 site packages sqlalchemy util concurrency py3k py line 121 in greenlet spawn value = await result File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 454 in prepare and execute self handle exception error File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 389 in handle exception self adapt connection handle exception error File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 684 in handle exception raise error File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 442 in prepare and execute self rows = await prepared stmt fetch parameters File usr local lib python3 9 site packages asyncpg prepared stmt py line 176 in fetch data = await self bind execute args 0 timeout File usr local lib python3 9 site packages asyncpg prepared stmt py line 241 in bind execute data status = await self do execute File usr local lib python3 9 site packages asyncpg prepared stmt py line 230 in do execute return await executor protocol File asyncpg protocol protocol pyx line 201 in bind execute asyncio exceptions TimeoutError```

Deepanshu Aggarwal · Answer

```INFO 10 0 120 44 15094 POST block documents filter HTTP 1 1 500 Internal Server Error ERROR Exception in ASGI application Traceback most recent call last File usr local lib python3 9 site packages uvicorn protocols http h11 impl py line 404 in run asgi result = await app type ignore func returns value File usr local lib python3 9 site packages uvicorn middleware proxy headers py line 78 in call return await scope receive send File usr local lib python3 9 site packages fastapi applications py line 270 in call await super call scope receive send File usr local lib python3 9 site packages starlette applications py line 124 in call await self middleware stack scope receive send File usr local lib python3 9 site packages starlette middleware errors py line 184 in call raise exc File usr local lib python3 9 site packages starlette middleware errors py line 162 in call await scope receive send File usr local lib python3 9 site packages starlette middleware cors py line 92 in call await self simple response scope receive send request headers=headers File usr local lib python3 9 site packages starlette middleware cors py line 147 in simple response await scope receive send File usr local lib python3 9 site packages starlette middleware exceptions py line 75 in call raise exc File usr local lib python3 9 site packages starlette middleware exceptions py line 64 in call await scope receive sender File usr local lib python3 9 site packages fastapi middleware asyncexitstack py line 21 in call raise e File usr local lib python3 9 site packages fastapi middleware asyncexitstack py line 18 in call await scope receive send File usr local lib python3 9 site packages starlette routing py line 680 in call await route handle scope receive send File usr local lib python3 9 site packages starlette routing py line 427 in handle await scope receive send File usr local lib python3 9 site packages fastapi applications py line 270 in call await super call scope receive send File usr local lib python3 9 site packages starlette applications py line 124 in call await self middleware stack scope receive send File usr local lib python3 9 site packages starlette middleware errors py line 184 in call raise exc File usr local lib python3 9 site packages starlette middleware errors py line 162 in call await scope receive send File usr local lib python3 9 site packages starlette middleware exceptions py line 75 in call raise exc File usr local lib python3 9 site packages starlette middleware exceptions py line 64 in call await scope receive sender File usr local lib python3 9 site packages fastapi middleware asyncexitstack py line 21 in call raise e File usr local lib python3 9 site packages fastapi middleware asyncexitstack py line 18 in call await scope receive send File usr local lib python3 9 site packages starlette routing py line 680 in call await route handle scope receive send File usr local lib python3 9 site packages starlette routing py line 275 in handle await scope receive send File usr local lib python3 9 site packages starlette routing py line 65 in app response = await func request File usr local lib python3 9 site packages prefect orion utilities server py line 101 in handle response scoped depends response = await default handler request File usr local lib python3 9 site packages fastapi routing py line 231 in app raw response = await run endpoint function File usr local lib python3 9 site packages fastapi routing py line 160 in run endpoint function return await dependant call values File usr local lib python3 9 site packages prefect orion api block documents py line 62 in read block documents result = await models block documents read block documents File usr local lib python3 9 site packages prefect orion database dependencies py line 117 in async wrapper return await fn args kwargs File usr local lib python3 9 site packages prefect orion models block documents py line 356 in read block documents result = await session execute File usr local lib python3 9 site packages sqlalchemy ext asyncio session py line 215 in execute result = await greenlet spawn File usr local lib python3 9 site packages sqlalchemy util concurrency py3k py line 126 in greenlet spawn result = context throw sys exc info File usr local lib python3 9 site packages sqlalchemy orm session py line 1712 in execute result = conn execute 20 statement params or execution options File usr local lib python3 9 site packages sqlalchemy engine base py line 1705 in execute 20 return meth self args 10style kwargs 10style execution options File usr local lib python3 9 site packages sqlalchemy sql elements py line 333 in execute on connection return connection execute clauseelement File usr local lib python3 9 site packages sqlalchemy engine base py line 1572 in execute clauseelement ret = self execute context File usr local lib python3 9 site packages sqlalchemy engine base py line 1943 in execute context self handle dbapi exception File usr local lib python3 9 site packages sqlalchemy engine base py line 2128 in handle dbapi exception util raise exc info 1 with traceback=exc info 2 File usr local lib python3 9 site packages sqlalchemy util compat py line 208 in raise raise exception File usr local lib python3 9 site packages sqlalchemy engine base py line 1900 in execute context self dialect do execute File usr local lib python3 9 site packages sqlalchemy engine default py line 736 in do execute cursor execute statement parameters File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 479 in execute self adapt connection await File usr local lib python3 9 site packages sqlalchemy util concurrency py3k py line 68 in await only return current driver switch awaitable File usr local lib python3 9 site packages sqlalchemy util concurrency py3k py line 121 in greenlet spawn value = await result File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 454 in prepare and execute self handle exception error File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 389 in handle exception self adapt connection handle exception error File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 684 in handle exception raise error File usr local lib python3 9 site packages sqlalchemy dialects postgresql asyncpg py line 442 in prepare and execute self rows = await prepared stmt fetch parameters File usr local lib python3 9 site packages asyncpg prepared stmt py line 176 in fetch data = await self bind execute args 0 timeout File usr local lib python3 9 site packages asyncpg prepared stmt py line 241 in bind execute data status = await self do execute File usr local lib python3 9 site packages asyncpg prepared stmt py line 230 in do execute return await executor protocol File asyncpg protocol protocol pyx line 201 in bind execute asyncio exceptions TimeoutError```

Mason Menges · Answer

Generally Speaking there are a number of factors that could be contributing to the timeouts You can increase the db timeout via the profile configuration or by setting an environment variable Specifically this setting `PREFECT ORION DATABASE TIMEOUT= 5 0 ` you can view available configuration with this command for prefect 2 `prefect config view show defaults`

Deepanshu Aggarwal · Answer

thanks < Mason Menges> increasing the timeout limits for api request and db timeout helped

Ankit · Answer

Hey < Deepanshu Aggarwal> what values of timeout worked for you Struggling with this for a while

Deepanshu Aggarwal · Answer

somewhere around 2 minutes

Ankit · Answer

thanks something else must be wrong My pods are going into CrashLoopBackOff with asyncio exceptions TimeoutError error

Deepanshu Aggarwal · Answer

are you self hosting orion as well or using prefect cloud

Ankit · Answer

self hosting

Deepanshu Aggarwal · Answer

try increasing the db timeouts

Ankit · Answer

yes just tried that and increased the prefect database connection timeouts further

Ankit · Answer

it s working fine right now thanks

Ankit · Answer

and pod just got killed again with the same error facepalm

Ankit · Answer

do you know which timeout should be increased for db timeouts

Deepanshu Aggarwal · Answer

```PREFECT ORION DATABASE TIMEOUT 600 PREFECT ORION DATABASE CONNECTION TIMEOUT 600```

Ankit · Answer

both are set to 600 but pod has restarted twice in 5 mins

Ankit · Answer

btw it s an external postgres db

Ankit · Answer

both of these are actually set to 3600