Hey there. We have been seeing this error during some flow runs `No heartbeat detected from the remo...

Jacolon Walker

10/20/2021, 9:48 PM

Hey there. We have been seeing this error during some flow runs

No heartbeat detected from the remote task; marking the run as failed.

. Any info on what is possibly causing this?

Jacolon Walker

10/20/2021, 9:48 PM

For more context we are hitting an external API and getting results then at some point the task seems to just stall out

Kevin Kho

10/20/2021, 9:58 PM

Hi @Jacolon Walker, I think this answer is a good starting point. You can also see this part of the docs for the configuration

Zach Schumacher

10/21/2021, 3:43 AM

this has been happening to us for awhile too, no fix has worked as of yet

Kevin Kho

10/21/2021, 3:59 AM

What version are you on? I think 0.15.4 propagates the error better to give us a better idea.

Kevin Kho

10/21/2021, 4:00 AM

Have you tried the heartbeat config? I tried replicating this by spinning up an API with a very long sleep call and couldn’t. Are you querying an API too?

Zach Schumacher

10/21/2021, 4:03 AM

yeah we tried the hearbeat config, no luck. we are on 0.15.4 on both the agent and the flows. not an api, a long running db query. confirmed its not a memory issue too.

Zach Schumacher

10/21/2021, 4:04 AM

ours is related to k8s autoscaler tho

Kevin Kho

10/21/2021, 4:10 AM

Gotcha. Will bring it up when I chat with the team

Zanie

10/21/2021, 4:13 AM

Agh. Are you getting any of our heartbeat failure logs?

Zach Schumacher

10/21/2021, 6:53 PM

i think what was happening is we had this in our job templates

Copy code

restartPolicy: OnFailure

Zach Schumacher

10/21/2021, 6:53 PM

switching this to Never to let prefect handle the retries for us, and I think this will resolve our issue

Zanie

10/21/2021, 6:58 PM

Glad to hear. I think that we should still handle multiple pods attempting to run the same flow better so if you have any heartbeat error logs that'd be helpful.

Zach Schumacher

10/21/2021, 6:59 PM

i don't have any interesting logs

Zach Schumacher

10/21/2021, 6:59 PM

i can share what was logged, one sec

Jacolon Walker

10/21/2021, 7:01 PM

We are on version 0.15.5 so should be good there

Jacolon Walker

10/21/2021, 7:02 PM

Re-running a flow now to see if it does the same thing

11 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.