https://prefect.io logo
Join Slack
Powered by
Could someone help me understand what is happening...
# prefect-community
j
Could someone help me understand what is happening with this flow_run?
Copy code
063bd788-bc0c-46f5-85e4-1092c2b19297
✅ 1
I changed the heartbeat mode and it has way way more memory than it needs but its still ocsasionally failing with;
Copy code
09:08:48
INFO
CloudTaskRunner
Task 'insert': Starting task run...
09:08:48
INFO
CloudTaskRunner
Task 'insert': Finished task run for task with final state: 'Pending'
09:08:48
INFO
CloudFlowRunner
Flow run RUNNING: terminal tasks are incomplete.
09:57:00
ERROR
prefect-server.ZombieKiller.TaskRun
No heartbeat detected from the remote task; marking the run as failed.
a
Joshua, we are aware that the heartbeat experience is not the best one, and it's on our radar to improve the UX in Prefect 2.0 - this topic explains the problem and shows some possible solutions, but that's all we have currently as a remedy to this issue
j
Yeah I have tried all of these - increased memory at least 2x what it needs - changed the heartbeat mode to 
threads
- not sure what else I can do?
Is this just something I am going to have to live with happening occasionally?
a
I guess so - all the solutions I'm aware of are in this Discourse topic and I'm aware they aren't perfect, but that's all we have - it's a difficult problem that we don't have the capacity to tackle in more depth in 1.0, but we'll do in 2.0
j
@Anna Geller do you have any sense of the downsides of turning off heartbeat altogether when I know the task is just long running? Would it be better to turn it off altogether at the flow level - or to use mutations to turn off before the task in question and then turn it back on after?
a
Afaik, the downside is that if this task run gets stuck, Zombie Killer won't try to kill and restart it
j
But if I am certain I don't want that behaviour then thats fine?
Thanks!
🙌 1
a
yup correct 👍
5 Views
Open in Slack
PreviousNext
Powered by