Channels
#prefect-community
Title
# prefect-community
j
Joshua Greenhalgh
06/13/2022, 10:40 AMCould someone help me understand what is happening with this flow_run?
Copy code
063bd788-bc0c-46f5-85e4-1092c2b19297✅ 1
I changed the heartbeat mode and it has way way more memory than it needs but its still ocsasionally failing with;
Copy code
09:08:48
INFO
CloudTaskRunner
Task 'insert': Starting task run...
09:08:48
INFO
CloudTaskRunner
Task 'insert': Finished task run for task with final state: 'Pending'
09:08:48
INFO
CloudFlowRunner
Flow run RUNNING: terminal tasks are incomplete.
09:57:00
ERROR
prefect-server.ZombieKiller.TaskRun
No heartbeat detected from the remote task; marking the run as failed.a
Anna Geller
06/13/2022, 11:02 AM
Joshua, we are aware that the heartbeat experience is not the best one, and it's on our radar to improve the UX in Prefect 2.0 - this topic explains the problem and shows some possible solutions, but that's all we have currently as a remedy to this issue
j
Joshua Greenhalgh
06/13/2022, 11:19 AMYeah I have tried all of these - increased memory at least 2x what it needs - changed the heartbeat mode to
threadsIs this just something I am going to have to live with happening occasionally?
a
Anna Geller
06/13/2022, 11:47 AM
I guess so - all the solutions I'm aware of are in this Discourse topic and I'm aware they aren't perfect, but that's all we have - it's a difficult problem that we don't have the capacity to tackle in more depth in 1.0, but we'll do in 2.0
j
Joshua Greenhalgh
07/04/2022, 1:44 PM@Anna Geller do you have any sense of the downsides of turning off heartbeat altogether when I know the task is just long running? Would it be better to turn it off altogether at the flow level - or to use mutations to turn off before the task in question and then turn it back on after?
a
Anna Geller
07/04/2022, 2:51 PM
Afaik, the downside is that if this task run gets stuck, Zombie Killer won't try to kill and restart it
j
Joshua Greenhalgh
07/04/2022, 3:03 PMBut if I am certain I don't want that behaviour then thats fine?
Thanks!
🙌 1
a
Anna Geller
07/04/2022, 5:33 PM
yup correct 👍
5 Views