h

    Hui Zheng

    1 year ago
    Hello, community, our prefect flow run gets the error of no heartbeat from time to time. It seems retrying, but actually nothing happen, the normal 20 mins run gets stuck there for hours. Any idea how to fix that?
    nicholas

    nicholas

    1 year ago
    Hi @Hui Zheng - task runs usually fail to send heartbeats when they're in resource-starved environments or when the process has been exited prematurely (something like calling
    sys.exit()
    ) - either of those sound like possible scenarios for you?
    h

    Hui Zheng

    1 year ago
    @nicholas thank you. I think it could be resource-starved environments. I am fine with runs fail to send heartbeats. My concern is that the retrying seems not working.
    How could I ensure that the retrying works properly? How do I fix the re-trying?
    @Dylan @Kyle Moon-Wright Thank you again for chatting today. Please see below the flow run that had this unresponsive stuck-in-retrying issuehttps://cloud.prefect.io/semios/flow-run/ee39d492-3c56-4f69-8721-b9eb9f32dd21?logId=
    Dylan

    Dylan

    1 year ago
    Thank you Hui!
    h

    Hui Zheng

    1 year ago
    @Dylan @Kyle Moon-Wright Hello, hope all is well.. this This
    failed-retry-when-no-heartbeat
    issue kept coming up. I would like to work with you to get to the bottom of it.
    <https://cloud.prefect.io/semios/flow-run/72082fcb-213b-4cc2-8d6e-d2a0b345cffe>
    <https://cloud.prefect.io/semios/flow-run/8bf9505f-5c0f-47a1-8cc4-cd9393d17bc5>
    <https://cloud.prefect.io/semios/flow-run/c85d6ad6-0074-4cdb-9a12-5b02ab100a11>
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Hey @Hui Zheng, I see that this flow is registered with Core version
    0.13.6
    . Would it be possible for you to upgrade to the current version (
    0.13.15
    ) and see if this behavior still occurs?
    Just want to be sure this issue hasn’t been addressed with previous versions. From there, we can definitely dig in with potential culprits I have in mind.
    h

    Hui Zheng

    1 year ago
    for sure, I could do that.
    we will have another production release next week, where I could upgrade the prefect flow version
    0.13.15
    . if the issue persists, I will let you know
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    ok great, please do 👍
    also if possible, please upgrade your Agent while you’re there! (to cross that culprit off the list 😉 )