Hi all I am getting a weird bug in production for a flow with a kubernetes run config that hangs when I redeploy it

Prefect Core Version: 1.2.4

I trigger a flow run from a github actions step and it just hangs (happened yesterday and the run state was still in progress for 2 days) even though the job does not exist I know the flow did not finish because I see that my ml model tracking service did not register a new model I triggered the flow manually and it successfully completed. This flow is also scheduled and completes. What could be causing this issue? intermittent read time out error on the agent:

2022-12-27T04:49:54+02:00 WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")'

FWIW - we get the same read timeout errors on our agent, haven’t been able to figure out how to address them yet

Hi Michael, does the flow run hang in a submitted, running, or scheduled state? If it hangs in a submitted or scheduled state, here are some articles with first steps for troubleshooting. • Why is my flow stuck in a Scheduled State? • Why is my flow stuck in a Submitted State? If it hangs in a running state, it could be the case that the kubernetes pod was terminated before the flow could complete. Just for clarification, how often does this happen? Does this occur every time a flow run is triggered from a github action?