https://prefect.io logo
Join Slack
Channels
Powered by Linen
Title
m

Matthew Roeschke

04/25/2022, 4:46 PM
Is this heartbeat configuration only applicable to Prefect cloud setups? https://docs.prefect.io/orchestration/concepts/services.html#heartbeat-configuration I have a (expected) long running job on a Prefect server setup trying to not have the zombie killer terminate the job: 
No heartbeat detected from the remote task; marking the run as failed.
k

Kevin Kho

04/25/2022, 4:47 PM
No, server should support it also.
m

Matthew Roeschke

04/25/2022, 4:48 PM
Should I specify 
PREFECT__SERVER__HEARTBEAT_MODE
instead 
PREFECT__CLOUD__HEARTBEAT_MODE
?
(as the env variable)
k

Kevin Kho

04/25/2022, 4:49 PM
No it would be cloud as well
✅ 1
m

Matthew Roeschke

04/25/2022, 9:03 PM
@Kevin Kho so I am also running this usng 
KubernetesRun
like 
KUBERNETES_RUNCONFIG = KubernetesRun(
    image=IMAGE,
    image_pull_secrets=[MANIFOLD_IMG_PULL_SECRET],
    image_pull_policy="Always",
    job_template_path=JOB_TEMPLATE_PATH,
    env={"PREFECT__CLOUD__HEARTBEAT_MODE": "thread"},
)
and I am still getting 
No heartbeat detected from the remote task; marking the run as failed.
k

Kevin Kho

04/25/2022, 9:05 PM
What is your task doing?
m

Matthew Roeschke

04/25/2022, 9:06 PM
It’s making http requests 
asynchronously
using Python’s built in 
async
framework
Is there a way to manually set this heartbeat timeout?
k

Kevin Kho

04/25/2022, 9:09 PM
Maybe if you are on server, but not for cloud. Is your compute (k8s pod or docker container) still going on? When that dies after using threaded heartbeats, it normally means the compute is dying or error-ing like running out of memory
m

Matthew Roeschke

04/25/2022, 9:13 PM
Yes, we are running a server setup and not cloud. Okay I guess that’s a reasonable guess. I’ll investigate that possibility
k

Kevin Kho

04/25/2022, 9:15 PM
I guess it might be hardcoded and not so configurable
m

Matthew Roeschke

04/25/2022, 9:20 PM
Gotcha. Would be cool to make this configurable in the future
k

Kevin Kho

04/25/2022, 9:28 PM
You can try turning it off and seeing if the task/flow completes?
m

Matthew Roeschke

04/25/2022, 9:30 PM
Thanks, this flow was starting to time out originally so that’s why I was investigating how to configure the heartbeat correctly in order to let the process run for a longer duration
View count: 44
Back to #prefect-communityJoin thread in Slack