Hello, community, our prefect flow run gets the er...
# prefect-community
h
Hello, community, our prefect flow run gets the error of no heartbeat from time to time. It seems retrying, but actually nothing happen, the normal 20 mins run gets stuck there for hours. Any idea how to fix that?
n
Hi @Hui Zheng - task runs usually fail to send heartbeats when they're in resource-starved environments or when the process has been exited prematurely (something like calling
sys.exit()
) - either of those sound like possible scenarios for you?
h
@nicholas thank you. I think it could be resource-starved environments. I am fine with runs fail to send heartbeats. My concern is that the retrying seems not working.
How could I ensure that the retrying works properly? How do I fix the re-trying?
@Dylan @Kyle Moon-Wright Thank you again for chatting today. Please see below the flow run that had this unresponsive stuck-in-retrying issue https://cloud.prefect.io/semios/flow-run/ee39d492-3c56-4f69-8721-b9eb9f32dd21?logId=
d
Thank you Hui!
h
@Dylan @Kyle Moon-Wright Hello, hope all is well.. this This
failed-retry-when-no-heartbeat
issue kept coming up. I would like to work with you to get to the bottom of it.
Copy code
<https://cloud.prefect.io/semios/flow-run/72082fcb-213b-4cc2-8d6e-d2a0b345cffe>
<https://cloud.prefect.io/semios/flow-run/8bf9505f-5c0f-47a1-8cc4-cd9393d17bc5>
<https://cloud.prefect.io/semios/flow-run/c85d6ad6-0074-4cdb-9a12-5b02ab100a11>
k
Hey @Hui Zheng, I see that this flow is registered with Core version
0.13.6
. Would it be possible for you to upgrade to the current version (
0.13.15
) and see if this behavior still occurs?
Just want to be sure this issue hasn’t been addressed with previous versions. From there, we can definitely dig in with potential culprits I have in mind.
h
for sure, I could do that.
we will have another production release next week, where I could upgrade the prefect flow version
0.13.15
. if the issue persists, I will let you know
k
ok great, please do 👍
also if possible, please upgrade your Agent while you’re there! (to cross that culprit off the list 😉 )