Hey all! I am new to prefect, recently want to run a scheduled prefect job, but seems its being killed from time to time by a prefect ZombieKiller task, with an error No heartbeat detected from the remote task; marking the run as failed. Any idea why this happens, and how to make sure my task always have a heartbeat could be detected by the prefect ZombieKiller? thank you.
a
Anna Geller
10/27/2021, 10:00 AM
@Xinyi Guo are your tasks memory intensive? It looks like your compute instance ran out of memory.
Prefect uses flow heartbeats to detect whether your flow run is still alive. Without it, if your flow run would die due to some infrastructure issues (Kubernetes pod deleted, instance ran out of memory, or crashed entirely), Prefect UI would be showing this flow run as Running.
x
Xinyi Guo
10/27/2021, 8:00 PM
@Anna Geller hi Anna, great thanks for your reply. Hmmm, I am sure the task is not crashed entirely, yeah, hit memory limit exceed could be one reason, I will monitoring it closely to see if that’s the case