Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
p
Payam Vaezi
11/30/2021, 7:10 PMWe have a flow task run that finished with the status of
RUNNINGSUCCESSFAILUREPENDINGLogs:
29 November 2021,12:17:54 prefect.CloudFlowRunner INFO "Beginning Flow run for '30021ac5-7d89-43c2-ae17-81b439d50d68'"
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'source': Starting task run..."
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'source': Finished task run for task with final state: 'Running'"
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'fork': Starting task run..."
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'fork': Finished task run for task with final state: 'Pending'"
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'preprocess': Starting task run..."
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'preprocess': Finished task run for task with final state: 'Pending'"
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'canary': Starting task run..."
29 November 2021,12:17:54 prefect.CloudTaskRunner INFO "Task 'canary': Finished task run for task with final state: 'Pending'"
29 November 2021,12:17:55 prefect.CloudTaskRunner INFO "Task 'concatenate': Starting task run..."
29 November 2021,12:17:55 prefect.CloudTaskRunner INFO "Task 'concatenate': Finished task run for task with final state: 'Pending'"
29 November 2021,12:17:55 prefect.CloudTaskRunner INFO "Task 'sink': Starting task run..."
29 November 2021,12:17:55 prefect.CloudTaskRunner INFO "Task 'sink': Finished task run for task with final state: 'Pending'"
29 November 2021,12:17:55 prefect.CloudFlowRunner INFO "Flow run RUNNING: terminal tasks are incomplete."
29 November 2021,12:28:27 prefect-server.ZombieKiller.TaskRun ERROR No heartbeat detected from the remote task; marking the run as failed.k
Kevin Kho
11/30/2021, 7:14 PMIs this task doing a long API call?
You can try changing the heartbeats to threads here and it might help
p
Payam Vaezi
11/30/2021, 7:18 PMThe source component is doing a snowflake query. I don’t believe it should be a long API call. I’ll try with different heartbeat mechanism to see if it solves the problem.
@William Schor
@David Harrington
@Kevin Kho still intermittently we get heartbeat failures even with threads. Do you have any other recommendations?
k
Kevin Kho
12/03/2021, 10:02 PMNormally the heartbeat not being detected is indicative that something happened to your flow, and then Prefect is just telling you that it lost communication. The most common reason I have seen for heartbeat failures is resource starvation. In Prefect 0.15.5 and up, the error message should be raised.
If you are confident that the Flow will succeed and you think Prefect is mistakenly marking it as failed (very rare), you can try turning heartbeats off for the Flow. Or you can put that task into it’s own subflow and turn off heartbeats on the subflow.
p
Payam Vaezi
12/10/2021, 4:32 PMThanks for the suggestions, we have tried increasing the resources on the job, but it still fails intermittently, do you recommend turning off heartbeat? What would be missing if we turn the heartbeat off?
k
Kevin Kho
12/10/2021, 4:35 PMIf you turn the heartbeat off and the compute dies, your Flow will be permanently seen as “Running” in the UI unless you mark it as Failed
p
Payam Vaezi
12/10/2021, 5:13 PMOk thanks for the info!