https://prefect.io logo
Join Slack
Powered by
Is this heartbeat configuration only applicable to...
# ask-community
m
Is this heartbeat configuration only applicable to Prefect cloud setups? https://docs.prefect.io/orchestration/concepts/services.html#heartbeat-configuration I have a (expected) long running job on a Prefect server setup trying to not have the zombie killer terminate the job: 
No heartbeat detected from the remote task; marking the run as failed.
k
No, server should support it also.
m
Should I specify 
PREFECT__SERVER__HEARTBEAT_MODE
instead 
PREFECT__CLOUD__HEARTBEAT_MODE
?
(as the env variable)
k
No it would be cloud as well
✅ 1
m
@Kevin Kho so I am also running this usng 
KubernetesRun
like
Copy code
KUBERNETES_RUNCONFIG = KubernetesRun(
    image=IMAGE,
    image_pull_secrets=[MANIFOLD_IMG_PULL_SECRET],
    image_pull_policy="Always",
    job_template_path=JOB_TEMPLATE_PATH,
    env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"},
)
and I am still getting 
No heartbeat detected from the remote task; marking the run as failed.
k
What is your task doing?
m
It’s making http requests 
asynchronously
using Python’s built in 
async
framework
Is there a way to manually set this heartbeat timeout?
k
Maybe if you are on server, but not for cloud. Is your compute (k8s pod or docker container) still going on? When that dies after using threaded heartbeats, it normally means the compute is dying or error-ing like running out of memory
m
Yes, we are running a server setup and not cloud. Okay I guess that’s a reasonable guess. I’ll investigate that possibility
k
I guess it might be hardcoded and not so configurable
m
Gotcha. Would be cool to make this configurable in the future
k
You can try turning it off and seeing if the task/flow completes?
m
Thanks, this flow was starting to time out originally so that’s why I was investigating how to configure the heartbeat correctly in order to let the process run for a longer duration
148 Views
Open in Slack
PreviousNext
Powered by