Hi everyone, I'm getting an error in all of my flows when running from Prefect Cloud. After ~9:50 mins (almost exactly) the execution of whatever task within the currently running subflow is running gets terminated. I see an error that's something like this:
Crash detected! Execution was cancelled by the runtime environment.
ERROR
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
self._run_once()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1869, in _run_once
event_list = self._selector.select(timeout)
File "/usr/local/lib/python3.9/selectors.py", line 469, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1613, in cancel_flow_run
raise TerminationSignal(signal=signal.SIGTERM)
prefect.exceptions.TerminationSignal
The the subflow keeps running indefinitely despite having a failed task, and the main flow fails (its seems similar to
https://github.com/PrefectHQ/prefect/issues/8481). This only seems to happen if the flow runs for 9 mins and 50 seconds, and it happens to whichever subflow/task is running at that time. Is there anything that might be causing this? Any help is greatly appreciated.