Hiya when using Prefect Cloud we ve experienced more than on

Question

Hiya when using Prefect Cloud we ve experienced more than once now a `No heartbeat detected` reported by the Zombie Killer but we can t see the task state set to failed Has anyone else experienced this

Anna Geller · Answer

Zombie Killer tries to kill any task run which doesn t emit heartbeats which may happen with long running jobs and or processes external to Prefect <https discourse prefect io t flow is failing with an error message no heartbeat detected from the remote task 79|This topic> discusses the issue in much more detail

Anna Geller · Answer

<https docs prefect io orchestration concepts services html zombie killer|10min is for Server 2 min is for Cloud> The `Zombie Killer` service is responsible for handling zombies which Prefect defines as tasks that claim to be running but haven t updated their heartbeat in the past 2 minutes Prefect Cloud or 10 minutes Prefect Server

Oliver Mannion · Answer

Thanks Anna I see the No heartbeat log line but my task did not move into the failed state Is this expected

Anna Geller · Answer

it might be depending on the use case I don t have enough information to tell Can you send your flow run ID and share the code for your flow looking more for Flow structure not your tasks

Oliver Mannion · Answer

Thanks Anna the flow run id is `37227baa a361 4e80 9333 ca1f15fb7104` This is the schematic if it helps

Oliver Mannion · Answer

```def process table tablename str maxgrab Parameter runids Parameter upstream task Optional Task = None gt Tuple Task Task retrieve runid task = retrieve runid tablename maxgrab runids run model task = run model tablename retrieve runid task if upstream task retrieve runid task set upstream upstream task return retrieve runid task run model task with Flow flow as flow maxgrab param = Parameter maxgrab default=maxgrab runids param = Parameter runids default=runids runids views = create runids views runids organisation run model organisation = process table organisation maxgrab param runids param runids organisation set upstream runids views sharded tablename = Constant invoice lineitem account statementline task run name= sharded tablename checkpoint=True result=PrefectResult apply map process table tablename=sharded tablename maxgrab=unmapped maxgrab param runids=unmapped runids param upstream task=unmapped run model organisation ```

Anna Geller · Answer

Thanks for sharing the ID the logs tell us that this run failed due to Kubernetes Error `pods prefect job 505449d8 rgl9r failed for this job` then one person from your team restarted the flow run and it then succeeded I don t see a Flow object in your flow code definition how do you define it Usually it s defined using ```with Flow yourflow as flow your task = your decorated task fn ``` but in the end everything seems successful thinking face is the issue solved now were you able to detect which task caused the heartbeat issues

Oliver Mannion · Answer

Thanks Anna yes we manually restarted it But we re trying to understand why the task didn t go into the Failed state and then retry automatically without us intervening Any ideas The flow is above at bottom of the snippet sorry the indentation isn t quite right

Oliver Mannion · Answer

We think task run `748e39b5 495e 4629 bd03 c7420cdfb581` run model is the task that had heartbeat issues and rightly so because the Kubernetes pod died

Anna Geller · Answer

gt why the task didn t go into the Failed state and then retry automatically without us intervening Any ideas do you know which task it was hard to say without knowing what this task does it may happen with some unclosed DB connections long running jobs and generally especially when running processes external to Prefect

Oliver Mannion · Answer

It s the `run model` task which runs dbt ``` task log stdout=True task run name= run model tablename max retries=3 retry delay=timedelta seconds=10 def run model tablename str runid str gt None logger = prefect context get logger save keyfile secret if runid dbt config = DbtShellTask profiles dir= return all=True log stderr=True stream output=True command = f dbt run models tag tablename vars runids runid command dbt config run command=command else f No runid for tablename dbt won t be launched ``` But unfortunately we have now lost all the logs and task state because we deleted the project disappointed I ll keep an eye out for it happening again

Anna Geller · Answer

It could be related to the fact that you run this shell task from another task you may try moving it out into the Flow scope as shown e g <https www prefect io blog flow of flows orchestrating elt with prefect and dbt |here> keep us posted