https://prefect.io logo
Title
n

Nick Coy

02/02/2023, 5:19 PM
Hi all, we just upgraded to prefect 2.7.10 and our agent is running on GKE, I have a flow that has finished in a completed state but then throws an error. This has happened the last two times I have tried to run the flow.
Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/local/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete self.run_forever() File "/usr/local/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1868, in _run_once event_list = self._selector.select(timeout) File "/usr/local/lib/python3.10/selectors.py", line 469, in select fd_event_list = self._selector.poll(timeout, max_ev) File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1614, in cancel_flow_run raise TerminationSignal(signal=signal.SIGTERM)
Finished in state Failed('Task run encountered an exception: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run\n    return loop.run_until_complete(main)\n  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete\n    self.run_forever()\n  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 603, in run_forever\n    self._run_once()\n  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1868, in _run_once\n    event_list = self._selector.select(timeout)\n  File "/usr/local/lib/python3.10/selectors.py", line 469, in select\n    fd_event_list = self._selector.poll(timeout, max_ev)\n  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1614, in cancel_flow_run\n    raise TerminationSignal(signal=signal.SIGTERM)\nprefect.exceptions.TerminationSignal\n\nDuring handling of the above exception, another exception occurred:\n\nAssertionError\n')
the flow is marked as complete
z

Zanie

02/02/2023, 5:21 PM
That’s peculiar can you share the full logs?
cc @Chris Pickett
n

Nick Coy

02/02/2023, 6:02 PM
@Zanie Here are the logs for one of the flow runs that failed. was marked as complete
w

Walter Cavinaw

02/02/2023, 6:41 PM
Btw @Christopher Boyd this (random Termination exceptions) was also one of the issues we kept seeing. @Nick Coy what version were you running before? Did you ever get the error with that version? I'd like to try it and see if that fixes our problems because our flows are crashing often.
z

Zanie

02/02/2023, 6:48 PM
Well here it looks like your pod was torn down? There’s a 404 indicating the pod was deleted and a SIGTERM is sent to the pod.
n

Nick Coy

02/02/2023, 6:52 PM
@Walter Cavinaw we were on 2.4.0 before. we did not see this error with that version
@Zanie I'll check that out, thanks
I noticed in the issue you linked we both have this warning
RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
z

Zanie

02/02/2023, 7:06 PM
The warning is meaningless for us
Not sure what the deal with it is, it’s on the backlog to investigate but it doesn’t change any behavior.
n

Nick Coy

02/02/2023, 7:23 PM
ok, thats good to know. Ill keep investigating to try and figure it out. Its just strange to me that it happens on a specific flow that we run