Channels
#prefect-community
Title
l
Lennert Van de Velde
10/20/2022, 12:06 PMHi all,
when doing a flow run with Concurrent flow runner, my run always ends af 204 tasks with a timeouterror. I have tried setting the PREFECT_ORION_DATABASE_TIMEOUT to 10, but it results in the same error. Database being used is SQlite.
c
Christopher Boyd
10/20/2022, 12:34 PMHi Lennert, would you mind moving your logs into the reply section so it doesn’t take up the full screen of the chat for other users?
l
Lennert Van de Velde
10/20/2022, 12:35 PM Copy code
Oct 20 13:29:35 [ __ ] sh[2178550]: /usr/local/lib/python3.9/site-packages/prefect/orion/schemas/states.py:147: UserWarning: State result is a 'CancelledError' type and is not safe to re-raise, it will be returned instead.
Oct 20 13:29:35 [ __ ] sh[2178550]: warnings.warn(
Oct 20 13:29:35 [ __ ] sh[2178550]: 11:29:04.963 | ERROR | Flow run 'ethereal-swift' - Crash detected! Request to http://[ __ ]:4200/api/task_runs/ failed.
Oct 20 13:29:35 [ __ ] sh[2178550]: 11:29:35.013 | ERROR | prefect.engine - Engine execution of flow run '3e101482-600c-4ded-82dd-475ea7ea0982' exited with unexpected exception
Oct 20 13:29:35 [ __ ] sh[2178550]: Traceback (most recent call last):
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_synchronization.py", line 38, in wait
Oct 20 13:29:35 [ __ ] sh[2178550]: await self._event.wait()
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1842, in wait
Oct 20 13:29:35 [ __ ] sh[2178550]: if await self._event.wait():
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/asyncio/locks.py", line 226, in wait
Oct 20 13:29:35 [ __ ] sh[2178550]: await fut
Oct 20 13:29:35 [ __ ] sh[2178550]: asyncio.exceptions.CancelledError
Oct 20 13:29:35 [ __ ] sh[2178550]: During handling of the above exception, another exception occurred:
Oct 20 13:29:35 [ __ ] sh[2178550]: Traceback (most recent call last):
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
Oct 20 13:29:35 [ __ ] sh[2178550]: yield
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_synchronization.py", line 38, in wait
Oct 20 13:29:35 [ __ ] sh[2178550]: await self._event.wait()
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
Oct 20 13:29:35 [ __ ] sh[2178550]: raise TimeoutError
Oct 20 13:29:35 [ __ ] sh[2178550]: TimeoutError
Oct 20 13:29:35 [ __ ] sh[2178550]: During handling of the above exception, another exception occurred:
Oct 20 13:29:35 [ __ ] sh[2178550]: Traceback (most recent call last):
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
Oct 20 13:29:35 [ __ ] sh[2178550]: yield
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
Oct 20 13:29:35 [ __ ] sh[2178550]: resp = await self._pool.handle_async_request(req)
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 234, in handle_async_request
Oct 20 13:29:35 [ __ ] sh[2178550]: raise exc
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 227, in handle_async_request
Oct 20 13:29:35 [ __ ] sh[2178550]: connection = await status.wait_for_connection(timeout=timeout)
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 34, in wait_for_connection
Oct 20 13:29:35 [ __ ] sh[2178550]: await self._connection_acquired.wait(timeout=timeout)
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_synchronization.py", line 38, in wait
Oct 20 13:29:35 [ __ ] sh[2178550]: await self._event.wait()
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
Oct 20 13:29:35 [ __ ] sh[2178550]: self.gen.throw(typ, value, traceback)
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
Oct 20 13:29:35 [ __ ] sh[2178550]: raise to_exc(exc)
Oct 20 13:29:35 [ __ ] sh[2178550]: httpcore.PoolTimeout
Oct 20 13:29:35 [ __ ] sh[2178550]: The above exception was the direct cause of the following exception:
Oct 20 13:29:35 [ __ ] sh[2178550]: Traceback (most recent call last):
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/prefect/client/orion.py", line 82, in with_injected_client
Oct 20 13:29:35 [ __ ] sh[2178550]: return await fn(*args, **kwargs)
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 306, in retrieve_flow_then_begin_flow_run
Oct 20 13:29:35 [ __ ] sh[2178550]: return await begin_flow_run(
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 368, in begin_flow_run
Oct 20 13:29:35 [ __ ] sh[2178550]: terminal_state = await orchestrate_flow_run(
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/contextlib.py", line 670, in __aexit__
Oct 20 13:29:35 [ __ ] sh[2178550]: raise exc_details[1]
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/contextlib.py", line 653, in __aexit__
Oct 20 13:29:35 [ __ ] sh[2178550]: cb_suppress = await cb(*exc_details)
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/contextlib.py", line 199, in __aexit__
Oct 20 13:29:35 [ __ ] sh[2178550]: await self.gen.athrow(typ, value, traceback)
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1360, in report_flow_run_crashes
Oct 20 13:29:35 [ __ ] sh[2178550]: await client.set_flow_run_state(
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/prefect/client/orion.py", line 1462, in set_flow_run_state
Oct 20 13:29:35 [ __ ] sh[2178550]: response = await <http://self._client.post|self._client.post>(
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1842, in post
Oct 20 13:29:35 [ __ ] sh[2178550]: return await self.request(
Oct 20 13:29:35 [ __ ] sh[2178550]: File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
Oct 20 13:29:36 [ __ ] sh[2178550]: return await self.send(request, auth=auth, follow_redirects=follow_redirects)c
Christopher Boyd
10/20/2022, 12:37 PMare these tasks all concurrent? are you just using the built-in sqlite that comes packaged with prefect?
if this is concurrent, you might want to consider deploying at least a standalone postgres, if you are running 204 concurrent tasks all trying to open connections simultaneously to a local sqlite, it could be resource / connection issues
do your other flows fails like this, or just this one? can you scale back and test a smaller amount
3 Views