Joish
07/26/2023, 7:15 AM### Error trace-stack ###
17:59:25.058 | ERROR | prefect.agent - An error occured while monitoring flow run 'fb822b8d-52ba-4781-a511-0a7bf3694308'. The flow run will not be marked as failed, but an issue may have occurred.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 485, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 308, in run
status_code = await run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 668, in _watch_job
for event in watch.stream(
File "/usr/local/lib/python3.10/site-packages/kubernetes/watch/watch.py", line 182, in stream
raise client.rest.ApiException(
kubernetes.client.exceptions.ApiException: (410)
Reason: Expired: too old resource version: 170074095 (170075178)
Has anyone encountered this issue before? Are there any suggestions on how to address it?
Issue 2:
To resolve the issue with the pending tasks, I attempted to retry them. While the retry worked properly in the UI, it ended up creating new jobs without cleaning up the old unfinished jobs in the K8s cluster. Strangely, the newly created jobs were cleaned up correctly once their respective tasks were completed.
Any insights or advice on dealing with this behaviour would be greatly appreciated.
Thanks in advanceDeceivious
07/26/2023, 8:26 AMJoish
07/27/2023, 5:02 PMDeceivious
07/27/2023, 5:03 PMJoish
07/27/2023, 5:28 PMTanishq Hooda
09/25/2023, 9:43 AMJoish
09/25/2023, 9:45 AMTanishq Hooda
09/25/2023, 9:46 AMJoish
09/25/2023, 9:47 AM