Did we ever solve <this issue> around timeout erro...
# ask-community
d
Did we ever solve this issue around timeout errors? I’m getting an issue with the same traceback when running semi-complicated logic in a Prefect 2.x
on_completion
hook. I have around 200 tasks and they each make updates to a shared progress store in the hook. I’m doing it this way until I upgrade to Prefect 3.0 as I cannot process tasks as they complete in 2.x
To add more detail here, I hit these errors
Copy code
await self.startup()
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asgi_lifespan/site-packages/asgi_lifespan/_manager.py", line 40, in startup
    await self._concurrency_backend.run_and_fail_after(
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asgi_lifespan/site-packages/asgi_lifespan/_concurrency/asyncio.py", line 46, in run_and_fail_after
    raise TimeoutError
Even if I do something super minimal like call
state.result()
in the hook The final error is
Copy code
04:50:52.961 | ERROR   | sqlalchemy.pool.impl.AsyncAdaptedQueuePool - Exception during reset or similar
Traceback (most recent call last):
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 757, in _finalize_fairy
    fairy._reset(pool, transaction_was_reset)
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 1016, in _reset
    pool._dialect.do_rollback(self)
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
    dbapi_connection.rollback()
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 731, in rollback
    self._handle_exception(error)
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 729, in rollback
    self.await_(self._transaction.rollback())
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 59, in await_only
    raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: <https://sqlalche.me/e/14/xd2s>)
04:50:52.963 | ERROR   | sqlalchemy.pool.impl.AsyncAdaptedQueuePool - Exception closing connection <AdaptedConnection <asyncpg.connection.Connection object at 0x7f6e17424e40>>
Traceback (most recent call last):
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 757, in _finalize_fairy
    fairy._reset(pool, transaction_was_reset)
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 1016, in _reset
    pool._dialect.do_rollback(self)
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
    dbapi_connection.rollback()
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 731, in rollback
    self._handle_exception(error)
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
    raise error
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 729, in rollback
    self.await_(self._transaction.rollback())
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 59, in await_only
    raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: <https://sqlalche.me/e/14/xd2s>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
    self._dialect.do_terminate(connection)
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 995, in do_terminate
    dbapi_connection.terminate()
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 752, in terminate
    self._connection.terminate()
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py", line 1344, in terminate
    self._abort()
  File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py", line 1371, in _abort
    self._protocol.abort()
  File "asyncpg/protocol/protocol.pyx", line 571, in asyncpg.protocol.protocol.BaseProtocol.abort
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 686, in abort
    self._force_close(None)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
04:50:52.974 | ERROR   | asyncio - Task was destroyed but it is pending!
task: <Task pending name='Task-3386' coro=<Connection._cancel() running at /app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py:1449> wait_for=<Future pending cb=[Task.task_wakeup()]>>
04:50:52.975 | ERROR   | asyncio - Task was destroyed but it is pending!
task: <Task pending name='Task-3464' coro=<Connection._cancel() running at /app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py:1449> wait_for=<Future pending cb=[Task.task_wakeup()]>>
04:50:52.982 | ERROR   | asyncio - Task was destroyed but it is pending!
task: <Task pending name='Task-3898' coro=<Connection._cancel() running at /app/src/python/studio/backend/py_image.binary.runfiles
This happens on both cached and uncached tasks, but it happens earlier for uncached tasks (due to reading from disk vs memory?) Is there some simple setting I can change to extend this timeout?