Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
t
Thomas Pedersen
11/09/2022, 10:05 AMWhen flows crash (according to their logs), Orion UI keeps showing the flows as running, and their run time keeps increasing.
If the flow isn't running on the agent, shouldn't it show as "Failed" and not running?
Still looking into what actually causes the crash - this is more about how Prefect handles them. We do have flows that runs successful as well, this is just a single flow or two crashing every time.
Here's the stack trace causing it - not sure it tells me much about what the issue is, but will look at it again later...
Crash details:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 88, in handle_async_request
raise ConnectionNotAvailable()
httpcore.ConnectionNotAvailable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1332, in report_task_run_crashes
yield
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1068, in begin_task_run
connect_error = await client.api_healthcheck()
File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 183, in api_healthcheck
await self._client.get("/health")
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1751, in get
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.10/site-packages/prefect/client/base.py", line 159, in send
await super().send(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 246, in handle_async_request
async with self._pool_lock:
File "/usr/local/lib/python3.10/site-packages/anyio/_core/_synchronization.py", line 130, in acquire
await event.wait()
File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledErrort
Tim Galvin
11/10/2022, 2:13 AMI have found this as well and I do not have a nice fix for it.
m
Mason Menges
11/11/2022, 10:30 PMHey @Thomas Pedersen @Tim Galvin we're tracking related issues around this here https://github.com/PrefectHQ/prefect/issues/7512 if you have some time we'd greatly appreciate adding your reports for these issues there
t
Thomas Pedersen
11/14/2022, 7:21 AM@Mason Menges, done - but if the agent never reports back to the server, should the server just leave the flows as running in the UI forever?
I feel like there need to be some kind of keep-alive reporting for long running flows, where the Agent keeps telling the server that the flow is active and running. If the server never hears anything from the Agent it then should assume that the flow is crashed or similar...