Hello 👋 We're using Prefect server and running our flows with the KubernetesAgent. Sometimes a flow run is running twice in parallel. After a bit of investigation I found this: The first flow run fails the heartbeat so the ZombieKiller retries the flow run (starting the parallel execution). But the first one is still running, it's not dead, it just didn't do the heartbeat because of long blocking operation. Any ideas how to prevent this? I don't even know how the heartbeat system works

No heartbeat detected from the remote task; retrying the run.

Hi @Lukas N.! Maybe turning off the Lazarus process will help you. You can do this under

Settings

in the UI. There is also information about these there.

Is there such a way to configure the heartbeat interval?

I don’t think so, only the ability to disable. Do you need them faster or slower? @Giovanni Giacco

Not necessarily. I’m struggling with long running task executing on Dask workers. Actually I disabled heartbeats and I have no more “heartbeat missing error” but my workers die with a KilledWorker exception anyway.

The heartbeat doesn’t keep the worker alive. It just checks if it’s up. I think the KilledWorker might be separate from Prefect? and it might be more about the processes are running on Dask

I thought when the worker is blocked by long I/O operations (processes on Dask read and process image from S3 buckets) it struggles with heartbeat checks but maybe I’m wrong. I’m a newbie of Prefect and Dask and I’m learning by doing.

If you’re using a Local agent, it may be possible to struggle, but otherwise the heartbeat is a lightweight process and it generally should be able to work (in the case of a Dask cluster)