Hi there, I'm trying to understand how the zombie killer decides to mark a task as failed and by extension how the heartbeats are actually sent. We use the local daemon with cloud which I believe forks a process per flow, who is doing the heartbeating there?

Hi @Kostas Chalikias! Task run heartbeats are sent by the executor and flow run heartbeats are sent by the agent and both are sent every 30 seconds; if a task or flow run misses four heartbeats in a row, it'll be marked as

Failed

by the zombie killer. You can turn this off in the Settings of your flow from the UI or by running this mutation:

mutation {
  disable_flow_heartbeat(input: { flow_id: "your-flow-id-here" }) {
    success
  }
}

So can a deadlock in the task cause this? What about a very busy task?

Yeah I think a deadlock could cause that but I'm not sure a task that's doing a lot would stop sending heartbeats. The only instance of that I've seen is when it runs into resource constraints

To elaborate a bit, I am seeing a task get marked as zombie on cloud when ran from our deployed daemon but I then run the flow locally on my laptop and it does seem to chug along, admittedly it looks like a very long task. Running locally is not ideal as there is latency to the rest of the infra and my laptop might go to sleep etc... So I want to understand how I can get that task to run properly on the server

Hi @Kostas Chalikias, I'll second that we generally see tasks marked as zombies when heartbeats fail to send due to resource issues. Without knowing more about your infrastructure, I can't give you an exact answer, but there is a workaround. If you're not overly concerned with heartbeats, you can disable heartbeats for your flow, either in the

Settings

tab for your flow in the UI or by running this GraphQL mutation. https://docs.prefect.io/orchestration/concepts/flows.html#toggling-heartbeats

Thanks, it might be worth ruling that out, I will give it a go