Is this heartbeat configuration only applicable to...
# ask-community
m
Is this heartbeat configuration only applicable to Prefect cloud setups? https://docs.prefect.io/orchestration/concepts/services.html#heartbeat-configuration I have a (expected) long running job on a Prefect server setup trying to not have the zombie killer terminate the job:
No heartbeat detected from the remote task; marking the run as failed.
k
No, server should support it also.
m
Should I specify
PREFECT__SERVER__HEARTBEAT_MODE
instead
PREFECT__CLOUD__HEARTBEAT_MODE
?
(as the env variable)
k
No it would be cloud as well
1
m
@Kevin Kho so I am also running this usng
KubernetesRun
like
Copy code
KUBERNETES_RUNCONFIG = KubernetesRun(
    image=IMAGE,
    image_pull_secrets=[MANIFOLD_IMG_PULL_SECRET],
    image_pull_policy="Always",
    job_template_path=JOB_TEMPLATE_PATH,
    env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"},
)
and I am still getting
No heartbeat detected from the remote task; marking the run as failed.
k
What is your task doing?
m
It’s making http requests
asynchronously
using Python’s built in
async
framework
Is there a way to manually set this heartbeat timeout?
k
Maybe if you are on server, but not for cloud. Is your compute (k8s pod or docker container) still going on? When that dies after using threaded heartbeats, it normally means the compute is dying or error-ing like running out of memory
m
Yes, we are running a server setup and not cloud. Okay I guess that’s a reasonable guess. I’ll investigate that possibility
k
I guess it might be hardcoded and not so configurable
m
Gotcha. Would be cool to make this configurable in the future
k
You can try turning it off and seeing if the task/flow completes?
m
Thanks, this flow was starting to time out originally so that’s why I was investigating how to configure the heartbeat correctly in order to let the process run for a longer duration
139 Views