Cameron Raynor
02/12/2023, 4:40 AMCrash detected! Execution was cancelled by the runtime environment.
ERROR
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
self._run_once()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1869, in _run_once
event_list = self._selector.select(timeout)
File "/usr/local/lib/python3.9/selectors.py", line 469, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1613, in cancel_flow_run
raise TerminationSignal(signal=signal.SIGTERM)
prefect.exceptions.TerminationSignal
The the subflow keeps running indefinitely despite having a failed task, and the main flow fails (its seems similar to https://github.com/PrefectHQ/prefect/issues/8481). This only seems to happen if the flow runs for 9 mins and 50 seconds, and it happens to whichever subflow/task is running at that time. Is there anything that might be causing this? Any help is greatly appreciated.Ryan Peden
02/12/2023, 6:11 AMCameron Raynor
02/12/2023, 4:57 PMAleksandr Liadov
02/14/2023, 4:32 PMCrash details:
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
File "/usr/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
event_list = self._selector.select(timeout)
File "/usr/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1637, in cancel_flow_run
raise TerminationSignal(signal=signal.SIGTERM)
prefect.exceptions.TerminationSignal
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1689, in report_task_run_crashes
yield
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1328, in begin_task_run
state = await orchestrate_task_run(
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1418, in orchestrate_task_run
resolved_parameters = await resolve_inputs(parameters)
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 1758, in resolve_inputs
return await run_sync_in_worker_thread(
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/opt/pysetup/.venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
asyncio.exceptions.CancelledError
Or it could be:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/engine.py", line 650, in orchestrate_flow_run
result = await run_sync(flow_call)
File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 165, in run_sync_in_interruptible_worker_thread
assert result is not NotSet
AssertionError
Any ideas, or suggestions where I should see?Cameron Raynor
02/16/2023, 2:18 AMAleksandr Liadov
02/16/2023, 8:11 AMCameron Raynor
02/16/2023, 7:50 PMkubectl config
file is causing the job to stop which is sending the termination signal to Prefect