Nick Coy
11/07/2022, 9:12 PMprefect.agent - An error occured while monitoring flow run <flow_run_id> The flow run will not be marked as failed, but an issue may have occurred.
Zanie
Nick Coy
11/07/2022, 9:48 PMZanie
Nick Coy
11/07/2022, 9:59 PM"ttlSecondsAfterFinished": 180
Zanie
Nick Coy
11/07/2022, 10:09 PMZanie
Nick Coy
11/07/2022, 10:20 PM08:32:42.927 | INFO | Task run 'run_bigquery_query-18a8523b-1' - hl-production-343314.media.pinterest_metrics_core
Nick Coy
11/07/2022, 10:37 PMFile "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 227, in _submit_run_and_capture_errors
result = await infrastructure.run(task_status=task_status)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 237, in run
return await run_sync_in_worker_thread(self._watch_job, job_name)
File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 432, in _watch_job
status_code=first_container_status.state.terminated.exit_code,
AttributeError: 'NoneType' object has no attribute 'exit_code'
Kristian Andersen Hole
01/23/2023, 7:47 PMAn error occured while monitoring flow run {uuid}. The flow run will not be marked as failed, but an issue may have occurred.This happens in about 1/20 runs. The flows then becomes stuck in the running state causing problems for other runs due to concurrency limits in our pipeline. Would be so grateful for any insight into this issue 🙂 Thanks
Zanie
Zanie
Kristian Andersen Hole
01/23/2023, 8:45 PMZanie
Zanie
Kristian Andersen Hole
01/23/2023, 10:50 PMZanie
Zanie