https://prefect.io logo
Title
m

Malavika S Menon

02/07/2023, 10:26 AM
Hi, my orion server threw this error
TimeoutError FlowRunNotifications took 5.071013 seconds to run, which is longer than its loop interval of 4 seconds.
Why exactly is this happening and how can this be avoided?
File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/services/loop_service.py", line 78, in start
    await self.run_once()
  File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/database/dependencies.py", line 117, in async_wrapper
    return await fn(*args, **kwargs)
  File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/services/flow_run_notifications.py", line 40, in run_once
    notifications = await db.get_flow_run_notifications_from_queue(
  File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/database/interface.py", line 280, in get_flow_run_notifications_from_queue
    return await self.queries.get_flow_run_notifications_from_queue(
  File "/root/.Envs/venv/lib/python3.8/site-packages/prefect/orion/database/query_components.py", line 394, in get_flow_run_notifications_from_queue
    result = await session.execute(notification_details_stmt)
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/ext/asyncio/session.py", line 215, in execute
    result = await greenlet_spawn(
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 128, in greenlet_spawn
    result = context.switch(value)
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1712, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2128, in _handle_dbapi_exception
    util.raise_(exc_info[1], with_traceback=exc_info[2])
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 479, in execute
    self._adapt_connection.await_(
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 68, in await_only
    return current.driver.switch(awaitable)
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 121, in greenlet_spawn
    value = await result
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 408, in _prepare_and_execute
    await adapt_connection._start_transaction()
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 716, in _start_transaction
    self._handle_exception(error)
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/root/.Envs/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 714, in _start_transaction
    await self._transaction.start()
  File "/root/.Envs/venv/lib/python3.8/site-packages/asyncpg/transaction.py", line 138, in start
    await self._connection.execute(query)
  File "/root/.Envs/venv/lib/python3.8/site-packages/asyncpg/connection.py", line 317, in execute
    return await self._protocol.query(query, timeout)
  File "asyncpg/protocol/protocol.pyx", line 338, in query
asyncio.exceptions.TimeoutError
FlowRunNotifications took 7.380488 seconds to run, which is longer than its loop interval of 4 seconds.
full traceback of error
Running prefect version 2.3.1
Any help would be appreciated
r

Rob Freedy

02/08/2023, 7:31 PM
Hey @Malavika S Menon!! Do you have any logs that look like the following?
Got {len(notifications)} notifications from queue.
This is from the FlowRunNotifications class, which is pulling one notification at a time for the reason stated in this comment. My best guess is there is a lot of notifications in the queue that are going over the timeout? You can open up an issue here for making that timeout variable an environment variable as a possible solution
🙌 1