Ben Muller
11/22/2022, 12:08 AMState message: Flow run encountered an exception. Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/client/orion.py", line 204, in api_healthcheck
await self._client.get("/health")
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1757, in get
return await self.request(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.9/site-packages/prefect/client/base.py", line 159, in send
await super().send(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1620, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1722, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 221, in handle_async_request
await self._attempt_to_acquire_connection(status)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 160, in _attempt_to_acquire_connection
status.set_connection(connection)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 22, in set_connection
assert self.connection is None
AssertionError
The above exception was the direct cause of the following exception:
RuntimeError: Cannot orchestrate task run '9cc3fc0a-ef27-4f5d-bba6-595b7c653c35'.
Tuoyi Zhao
11/22/2022, 12:34 AMBen Muller
11/22/2022, 8:28 AMAnna Geller
11/22/2022, 11:52 AMJosh Paulin
11/22/2022, 6:47 PMDaskTaskRunner
(more than 500 in the group). Partway though adding them I’ll see the error and the flow will crash.
This is using the local dask cluster option. Crashes don’t happen on every run.Anna Geller
11/22/2022, 9:27 PMBen Muller
11/23/2022, 5:57 AM.submit
and the ConcurrentTaskRunner
.
Ak
The exact same way of running things with .map used to work with Prefect 1.0 and now it just breaks. It is not ideal.
Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/client/orion.py", line 204, in api_healthcheck
await self._client.get("/health")
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1757, in get
return await self.request(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.9/site-packages/prefect/client/base.py", line 159, in send
await super().send(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1620, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1648, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1722, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 221, in handle_async_request
await self._attempt_to_acquire_connection(status)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 160, in _attempt_to_acquire_connection
status.set_connection(connection)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 22, in set_connection
assert self.connection is None
AssertionError
The above exception was the direct cause of the following exception:
RuntimeError: Cannot orchestrate task run 'ccdabf35-19ae-4fc3-91f9-2af74acf111b'. Failed to connect to API at <https://api.prefect.cloud/api/accounts/a0af308b-5af5-4edf-a50e-945f515efc16/workspaces/8e15f190-bb28-45fa-a1e8-b3335e28b718/>.
Is there anyway I can handle these errors in my code or is this just a downside of using async?Anna Geller
11/23/2022, 1:25 PMBen Muller
11/23/2022, 6:19 PMAnna Geller
11/23/2022, 6:49 PMBen Muller
11/23/2022, 8:48 PMAnna Geller
11/23/2022, 9:50 PMalvin goh
11/24/2022, 10:08 AMDeepanshu Aggarwal
11/24/2022, 11:36 AMalvin goh
11/24/2022, 12:39 PMDeepanshu Aggarwal
11/24/2022, 1:14 PMBen Muller
11/24/2022, 3:04 PMPeyton Runyan
11/24/2022, 3:42 PMDeepanshu Aggarwal
11/25/2022, 6:10 AMGiuliano Mega
11/25/2022, 6:11 PMDeepanshu Aggarwal
11/26/2022, 5:20 AMGiuliano Mega
12/01/2022, 9:14 PMPeyton Runyan
12/01/2022, 9:40 PMhttpcore
version?Ben Muller
12/01/2022, 9:47 PMPeyton Runyan
12/01/2022, 9:48 PMDeepanshu Aggarwal
12/02/2022, 7:44 AM