Hi! I'm having an issue on Prefect Cloud where a f...
# ask-community
j
Hi! I'm having an issue on Prefect Cloud where a flow appears as failed, despite no tasks having returned an error. The weirdest thing is the flow lasted 2 minutes, but one task is still marked as ongoing (blue) with its timer going up. What I don't understand is if there is still an active task, wouldn't the flow also be active past 2 minutes?
k
Hey @Jérémy Trudel, could you show me the schematic maybe so I can get a better picture of what’s going on?
j
Relaunched the same flow without modification and the task in question is now an error with "No heartbeat detected from the remote task; marking the run as failed." So far the only information I found to prevent this was to add the following line to my kubernetes template but it's not working: "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
d
Hi Jérémy! When a flow is started Prefect will also spin up a separate heartbeat process that tells the server that a flow is still executing and the machine hasn't simply crashed. In rare cases, we've observed the heartbeat process killed by the system independently of the executor process, and Prefect's server will consider the process dead after some time. Why this happens is still a bit mysterious to us, but our guess is that sometimes under memory pressure the heartbeat process will be terminated by the OOM killer. You can consider trying one of two solutions: increasing the memory on the cluster, or switching to using threaded heartbeats: documentation for that can be found [here](https://docs.prefect.io/orchestration/concepts/services.html#heartbeat-configuration)