https://prefect.io logo
#prefect-community
Title
# prefect-community
k

Keith

07/08/2022, 2:00 PM
Hi All! I have been loving my experience with Prefect but now that I am deploying to Kubernetes (GKE) I am having the all too common problem of it running fine locally but not in the cloud. Initially I was running into
OOMError
but got around that by increasing the memory requested from k8s. Now I am stuck on having the ZombieKiller stop tasks that should be reporting back status since they continue to produce logs with the message
No heartbeat detected from the remote task; marking the run as failed.
I have attempted to set the
HEARTBEAT_MODE
to
thread
and
off
via the
config.toml
file as well as in the
KubernetesRun
environment variables but no matter the combination I setup I still run into heartbeat errors killing a process. I am curious if there is a way to send a heartbeat from within the code base or if there is another approach I should take to get around my time out issue. Cheers!
1
a

Anna Geller

07/08/2022, 2:18 PM
Sorry to see you struggle with this but we don't really have a solution for this in Prefect 1.0 as of now, we'll work on improving that in 2.0 for now some solutions you may try are listed here: https://discourse.prefect.io/t/flow-is-failing-with-an-error-message-no-heartbeat-detected-from-the-remote-task/79
k

Keith

07/08/2022, 4:50 PM
Thank you for the response! Have definitely used that page and a couple of your posts in Discourse to help troubleshoot this issue.
🙌 1
Will look into ways to break up the processing a bit more to see if there are other viable methods.
👍 1
I was attempting to run this pipeline for a full days worth of data, I believe what was happening is that some of the tasks to collect the data were taking longer than 5 minutes. I ended up changing to running this pipeline every hour and now everything is running great and everything runs in ~30 seconds each run!
Is there a general guideline on when/if I should migrate to Prefect 2.0? I see that it is in beta but curious if it is stable enough to try to migrate now.
a

Anna Geller

07/11/2022, 7:44 PM
about when: it's up to you, you can start migrating already about how, check out https://discourse.prefect.io/tag/migration-guide + more docs are in the works
🙌 1
r

Rajeshwar Agrawal

07/13/2022, 10:09 AM
so is this already fixed in prefect 2.0?
a

Anna Geller

07/13/2022, 10:55 AM
no, it isn't - you asked 2 separate questions 1. my first answer is to the heartbeat question 2. my second answer is for migration to the 2.0 question
👍 1