Hi, we're having an issue with prefect cloud and the heartbeat mechanism marking a task as failed, but not marking the flow as failed. We are running flows using an EKS cluster that uses spot instances, so we rely on the heartbeat mechanism to know to retry flows that die because the underlying node is reclaimed by AWS. For flows where we're encountering the issue(see flow

9a3a44b8-efeb-48bf-b07c-22a90ca9b63e

), the log messages state:

No heartbeat detected from the remote task; marking the run as failed.

for some task, but the flow is still in a running state. The issue is intermittent,

1f570a06-56c8-46e4-acf5-3e0cf6d12b75

is a flow where the heartbeat marked the same task as failed and it also successfully marked the flow as failed. Any insight into what may be going on here would be much appreciated!

Is there Dask involved here? This different behavior is the same flow, just different runs right?

We removed the dask executor from this flow and it is still happening. Yep, it is the same flow, but it happens at random from run to run.