Hi everyone, I'm getting an error in all of my flows when running from Prefect Cloud. After ~9:50 mi...
c

Cameron Raynor

over 2 years ago
Hi everyone, I'm getting an error in all of my flows when running from Prefect Cloud. After ~9:50 mins (almost exactly) the execution of whatever task within the currently running subflow is running gets terminated. I see an error that's something like this:
Crash detected! Execution was cancelled by the runtime environment.

ERROR
Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1869, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/lib/python3.9/selectors.py", line 469, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1613, in cancel_flow_run
    raise TerminationSignal(signal=signal.SIGTERM)
prefect.exceptions.TerminationSignal
The the subflow keeps running indefinitely despite having a failed task, and the main flow fails (its seems similar to https://github.com/PrefectHQ/prefect/issues/8481). This only seems to happen if the flow runs for 9 mins and 50 seconds, and it happens to whichever subflow/task is running at that time. Is there anything that might be causing this? Any help is greatly appreciated.
1