Hi All! I have been loving my experience with Prefect but now that I am deploying to Kubernetes (GKE) I am having the all too common problem of it running fine locally but not in the cloud. Initially I was running into

OOMError

but got around that by increasing the memory requested from k8s. Now I am stuck on having the ZombieKiller stop tasks that should be reporting back status since they continue to produce logs with the message

No heartbeat detected from the remote task; marking the run as failed.

I have attempted to set the

HEARTBEAT_MODE

to

thread

and

off

via the

config.toml

file as well as in the

KubernetesRun

environment variables but no matter the combination I setup I still run into heartbeat errors killing a process. I am curious if there is a way to send a heartbeat from within the code base or if there is another approach I should take to get around my time out issue. Cheers!

Sorry to see you struggle with this but we don't really have a solution for this in Prefect 1.0 as of now, we'll work on improving that in 2.0 for now some solutions you may try are listed here: https://discourse.prefect.io/t/flow-is-failing-with-an-error-message-no-heartbeat-detected-from-the-remote-task/79

Thank you for the response! Have definitely used that page and a couple of your posts in Discourse to help troubleshoot this issue.

about when: it's up to you, you can start migrating already about how, check out https://discourse.prefect.io/tag/migration-guide + more docs are in the works

so is this already fixed in prefect 2.0?

no, it isn't - you asked 2 separate questions 1. my first answer is to the heartbeat question 2. my second answer is for migration to the 2.0 question