Hello everyone 👋. We're hosting prefect server in k8s and also running our flows there using the k8s agent. Everything works great, we have a Slack notification (state handler) setup to notify us when something goes wrong. However, when the pod running the flow is killed externally (e.g. due to OOM), we get no slack notification. This makes sense because the notifications happens from within the flow pod. Is there a way how to send these kind of notifications from other Prefect server components (towel) ? Or should this case be handled on kubernetes cluster with OOMkill alerting?

Hello Lukas 👋 Unfortunately, adding slack notification state handler in towel is not on Prefect radar, but I will share your idea with Prefect team!

Thank you 🙂