https://prefect.io logo
#prefect-community
Title
# prefect-community
m

Michael Levenson

12/27/2022, 4:41 PM
Hi all I am getting a weird bug in production for a flow with a kubernetes run config that hangs when I redeploy it
Prefect Core Version: 1.2.4
I trigger a flow run from a github actions step and it just hangs (happened yesterday and the run state was still in progress for 2 days) even though the job does not exist I know the flow did not finish because I see that my ml model tracking service did not register a new model I triggered the flow manually and it successfully completed. This flow is also scheduled and completes. What could be causing this issue? intermittent read time out error on the agent:
Copy code
2022-12-27T04:49:54+02:00 WARNING:urllib3.connectionpool:Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)")'
m

Marwan Sarieddine

12/27/2022, 6:01 PM
FWIW - we get the same read timeout errors on our agent, haven’t been able to figure out how to address them yet
👀 1
b

Bianca Hoch

12/27/2022, 6:11 PM
Hi Michael, does the flow run hang in a submitted, running, or scheduled state? If it hangs in a submitted or scheduled state, here are some articles with first steps for troubleshooting. • Why is my flow stuck in a Scheduled State?Why is my flow stuck in a Submitted State? If it hangs in a running state, it could be the case that the kubernetes pod was terminated before the flow could complete. Just for clarification, how often does this happen? Does this occur every time a flow run is triggered from a github action?
3 Views