Hi all we just upgraded to prefect 2 7 10 and our agent is r

Question

Hi all we just upgraded to prefect 2 7 10 and our agent is running on GKE I have a flow that has finished in a completed state but then throws an error This has happened the last two times I have tried to run the flow

Nick Coy · Answer

``` Traceback most recent call last File usr local lib python3 10 asyncio runners py line 44 in run return loop run until complete main File usr local lib python3 10 asyncio base events py line 636 in run until complete self run forever File usr local lib python3 10 asyncio base events py line 603 in run forever self run once File usr local lib python3 10 asyncio base events py line 1868 in run once event list = self selector select timeout File usr local lib python3 10 selectors py line 469 in select fd event list = self selector poll timeout max ev File usr local lib python3 10 site packages prefect engine py line 1614 in cancel flow run raise TerminationSignal signal=signal SIGTERM ```

Nick Coy · Answer

```Finished in state Failed Task run encountered an exception Traceback most recent call last n File usr local lib python3 10 asyncio runners py line 44 in run n return loop run until complete main n File usr local lib python3 10 asyncio base events py line 636 in run until complete n self run forever n File usr local lib python3 10 asyncio base events py line 603 in run forever n self run once n File usr local lib python3 10 asyncio base events py line 1868 in run once n event list = self selector select timeout n File usr local lib python3 10 selectors py line 469 in select n fd event list = self selector poll timeout max ev n File usr local lib python3 10 site packages prefect engine py line 1614 in cancel flow run n raise TerminationSignal signal=signal SIGTERM nprefect exceptions TerminationSignal n nDuring handling of the above exception another exception occurred n nAssertionError n ```

Nick Coy · Answer

the flow is marked as complete

Zanie · Answer

That s peculiar can you share the full logs

Zanie · Answer

cc < Chris Pickett>

Nick Coy · Answer

< Zanie> Here are the logs for one of the flow runs that ~failed ~ was marked as complete

Walter Cavinaw · Answer

Btw < Christopher Boyd> this random Termination exceptions was also one of the issues we kept seeing < Nick Coy> what version were you running before Did you ever get the error with that version I d like to try it and see if that fixes our problems because our flows are crashing often

Zanie · Answer

Well here it looks like your pod was torn down There s a 404 indicating the pod was deleted and a SIGTERM is sent to the pod

Zanie · Answer

Possibly related <https github com PrefectHQ prefect issues 8243>

Nick Coy · Answer

< Walter Cavinaw> we were on 2 4 0 before we did not see this error with that version

Nick Coy · Answer

< Zanie> I ll check that out thanks

Nick Coy · Answer

I noticed in the issue you linked we both have this warning ```RuntimeWarning prefect engine found in sys modules after import of package prefect but prior to execution of prefect engine this may result in unpredictable behaviour```

Zanie · Answer

The warning is meaningless for us

Zanie · Answer

Not sure what the deal with it is it s on the backlog to investigate but it doesn t change any behavior

Nick Coy · Answer

ok thats good to know Ill keep investigating to try and figure it out Its just strange to me that it happens on a specific flow that we run