Hello prefect community 🙂, We're using prefect with K8S. Sometime some tasks fail because the pods that run them get killed (for OOM or anything else, doesn't matter), which leads to a "pod missed heartbeat..." message. The issue is that when a flow/task fails like this, the retry mechanism and the slack handler doesn't work. Is there a way to retry tasks on this kind of failure or/and to get any notification about the status ?

Hi @Thomas Opsomer, so in this case, the compute literally dies so there is nothing that can retry that task. The notification part is doable though. You can use Automations in Prefect Cloud where Prefect will be the one responsible for sending an alert in the event it loses communication with you flow

Ah ok, that makes sense ! For the notification part, is there a special "state" that defines the case where the communiation with the flow is lost ?

No for current Prefect, but it will be in Orion (Prefect 2.0)!