Malavika S Menon
02/07/2023, 10:26 AMTimeoutError FlowRunNotifications took 5.071013 seconds to run, which is longer than its loop interval of 4 seconds.
Why exactly is this happening and how can this be avoided?File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/services/loop_service.py", line 78, in start
await self.run_once()
File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/database/dependencies.py", line 117, in async_wrapper
return await fn(*args, **kwargs)
File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/services/flow_run_notifications.py", line 40, in run_once
notifications = await db.get_flow_run_notifications_from_queue(
File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/database/interface.py", line 280, in get_flow_run_notifications_from_queue
return await self.queries.get_flow_run_notifications_from_queue(
File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/database/query_components.py", line 394, in get_flow_run_notifications_from_queue
result = await session.execute(notification_details_stmt)
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/session.py", line 215, in execute
result = await greenlet_spawn(
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 128, in greenlet_spawn
result = context.switch(value)
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1712, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
return connection._execute_clauseelement(
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
ret = self._execute_context(
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self._handle_dbapi_exception(
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2128, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
self._adapt_connection.await_(
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
return current.driver.switch(awaitable)
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
value = await result
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 408, in _prepare_and_execute
await adapt_connection._start_transaction()
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 716, in _start_transaction
self._handle_exception(error)
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
raise error
File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 714, in _start_transaction
await self._transaction.start()
File "/root/.Envs/venv/lib/python3.8/site-packages/asyncpg/transaction.py", line 138, in start
await self._connection.execute(query)
File "/root/.Envs/venv/lib/python3.8/site-packages/asyncpg/connection.py", line 317, in execute
return await self._protocol.query(query, timeout)
File "asyncpg/protocol/protocol.pyx", line 338, in query
asyncio.exceptions.TimeoutError
FlowRunNotifications took 7.380488 seconds to run, which is longer than its loop interval of 4 seconds.
full traceback of errorRob Freedy
02/08/2023, 7:31 PMGot {len(notifications)} notifications from queue.
This is from the FlowRunNotifications class, which is pulling one notification at a time for the reason stated in this comment. My best guess is there is a lot of notifications in the queue that are going over the timeout? You can open up an issue here for making that timeout variable an environment variable as a possible solution