Derek
09/17/2024, 1:13 AMon_completion
hook. I have around 200 tasks and they each make updates to a shared progress store in the hook. I’m doing it this way until I upgrade to Prefect 3.0 as I cannot process tasks as they complete in 2.xDerek
09/17/2024, 4:52 AMawait self.startup()
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asgi_lifespan/site-packages/asgi_lifespan/_manager.py", line 40, in startup
await self._concurrency_backend.run_and_fail_after(
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asgi_lifespan/site-packages/asgi_lifespan/_concurrency/asyncio.py", line 46, in run_and_fail_after
raise TimeoutError
Even if I do something super minimal like call state.result()
in the hook
The final error is
04:50:52.961 | ERROR | sqlalchemy.pool.impl.AsyncAdaptedQueuePool - Exception during reset or similar
Traceback (most recent call last):
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 757, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 1016, in _reset
pool._dialect.do_rollback(self)
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 731, in rollback
self._handle_exception(error)
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
raise error
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 729, in rollback
self.await_(self._transaction.rollback())
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 59, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: <https://sqlalche.me/e/14/xd2s>)
04:50:52.963 | ERROR | sqlalchemy.pool.impl.AsyncAdaptedQueuePool - Exception closing connection <AdaptedConnection <asyncpg.connection.Connection object at 0x7f6e17424e40>>
Traceback (most recent call last):
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 757, in _finalize_fairy
fairy._reset(pool, transaction_was_reset)
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 1016, in _reset
pool._dialect.do_rollback(self)
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
dbapi_connection.rollback()
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 731, in rollback
self._handle_exception(error)
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 684, in _handle_exception
raise error
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 729, in rollback
self.await_(self._transaction.rollback())
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 59, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: <https://sqlalche.me/e/14/xd2s>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/pool/base.py", line 260, in _close_connection
self._dialect.do_terminate(connection)
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 995, in do_terminate
dbapi_connection.terminate()
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_sqlalchemy/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 752, in terminate
self._connection.terminate()
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py", line 1344, in terminate
self._abort()
File "/app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py", line 1371, in _abort
self._protocol.abort()
File "asyncpg/protocol/protocol.pyx", line 571, in asyncpg.protocol.protocol.BaseProtocol.abort
File "/usr/lib/python3.10/asyncio/selector_events.py", line 686, in abort
self._force_close(None)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
04:50:52.974 | ERROR | asyncio - Task was destroyed but it is pending!
task: <Task pending name='Task-3386' coro=<Connection._cancel() running at /app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py:1449> wait_for=<Future pending cb=[Task.task_wakeup()]>>
04:50:52.975 | ERROR | asyncio - Task was destroyed but it is pending!
task: <Task pending name='Task-3464' coro=<Connection._cancel() running at /app/src/python/studio/backend/py_image.binary.runfiles/pip_deps_asyncpg/site-packages/asyncpg/connection.py:1449> wait_for=<Future pending cb=[Task.task_wakeup()]>>
04:50:52.982 | ERROR | asyncio - Task was destroyed but it is pending!
task: <Task pending name='Task-3898' coro=<Connection._cancel() running at /app/src/python/studio/backend/py_image.binary.runfiles
Derek
09/17/2024, 4:52 AM