Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all, we just upgraded to prefect 2.7.10 and our agent is running on GKE, I have a flow that has finished in a completed state but then throws an error. This has happened the last two times I have tried to run the flow.

```
Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/local/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete self.run_forever() File "/usr/local/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1868, in _run_once event_list = self._selector.select(timeout) File "/usr/local/lib/python3.10/selectors.py", line 469, in select fd_event_list = self._selector.poll(timeout, max_ev) File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1614, in cancel_flow_run raise TerminationSignal(signal=signal.SIGTERM)```

```Finished in state Failed('Task run encountered an exception: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run\n    return loop.run_until_complete(main)\n  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete\n    self.run_forever()\n  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 603, in run_forever\n    self._run_once()\n  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 1868, in _run_once\n    event_list = self._selector.select(timeout)\n  File "/usr/local/lib/python3.10/selectors.py", line 469, in select\n    fd_event_list = self._selector.poll(timeout, max_ev)\n  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1614, in cancel_flow_run\n    raise TerminationSignal(signal=signal.SIGTERM)\nprefect.exceptions.TerminationSignal\n\nDuring handling of the above exception, another exception occurred:\n\nAssertionError\n')```

That’s peculiar can you share the full logs?

downloaded-logs-20230202-095635.csv

<@U01CEUST9B5> Here are the logs for one of the flow runs that ~failed.~  was marked as complete

Btw <@U03FKQX1TH9> this (random Termination exceptions) was also one of the issues we kept seeing. <@U0242MQPUU9> what version were you running before? Did you ever get the error with that version? I'd like to try it and see if that fixes our problems because our flows are crashing often.

Well here it looks like your pod was torn down? There’s a 404 indicating the pod was deleted and a SIGTERM is sent to the pod.

Possibly related <https://github.com/PrefectHQ/prefect/issues/8243>

<@U03CYASS64C> we were on 2.4.0 before. we did not see this error with that version

<@U01CEUST9B5> I'll check that out, thanks

I noticed in the issue you linked we both have this warning
```RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour```


Not sure what the deal with it is, it’s on the backlog to investigate but it doesn’t change any behavior.

ok, thats good to know. Ill keep investigating to try and figure it out. Its just strange to me that it happens on a specific flow that we run