https://prefect.io logo
h

Hui Zheng

11/05/2020, 6:06 PM
Hello, community, our prefect flow run gets the error of no heartbeat from time to time. It seems retrying, but actually nothing happen, the normal 20 mins run gets stuck there for hours. Any idea how to fix that?
n

nicholas

11/05/2020, 6:08 PM
Hi @Hui Zheng - task runs usually fail to send heartbeats when they're in resource-starved environments or when the process has been exited prematurely (something like calling
sys.exit()
) - either of those sound like possible scenarios for you?
h

Hui Zheng

11/06/2020, 7:08 PM
@nicholas thank you. I think it could be resource-starved environments. I am fine with runs fail to send heartbeats. My concern is that the retrying seems not working.
How could I ensure that the retrying works properly? How do I fix the re-trying?
@Dylan @Kyle Moon-Wright Thank you again for chatting today. Please see below the flow run that had this unresponsive stuck-in-retrying issue https://cloud.prefect.io/semios/flow-run/ee39d492-3c56-4f69-8721-b9eb9f32dd21?logId=
d

Dylan

11/11/2020, 6:33 PM
Thank you Hui!
h

Hui Zheng

11/16/2020, 10:04 PM
@Dylan @Kyle Moon-Wright Hello, hope all is well.. this This
failed-retry-when-no-heartbeat
issue kept coming up. I would like to work with you to get to the bottom of it.
Copy code
<https://cloud.prefect.io/semios/flow-run/72082fcb-213b-4cc2-8d6e-d2a0b345cffe>
<https://cloud.prefect.io/semios/flow-run/8bf9505f-5c0f-47a1-8cc4-cd9393d17bc5>
<https://cloud.prefect.io/semios/flow-run/c85d6ad6-0074-4cdb-9a12-5b02ab100a11>
k

Kyle Moon-Wright

11/16/2020, 10:14 PM
Hey @Hui Zheng, I see that this flow is registered with Core version
0.13.6
. Would it be possible for you to upgrade to the current version (
0.13.15
) and see if this behavior still occurs?
Just want to be sure this issue hasn’t been addressed with previous versions. From there, we can definitely dig in with potential culprits I have in mind.
h

Hui Zheng

11/16/2020, 10:15 PM
for sure, I could do that.
we will have another production release next week, where I could upgrade the prefect flow version
0.13.15
. if the issue persists, I will let you know
k

Kyle Moon-Wright

11/16/2020, 10:16 PM
ok great, please do 👍
also if possible, please upgrade your Agent while you’re there! (to cross that culprit off the list 😉 )