hi prefect community! we are using prefect notifications in the prefect cloud to detect late flows in order to check if our agents are healthy. In case a run is late we send a notification via slack to be informed. Unfortunately we encountered some false positives, so flows were marked as LATE but the agent is healthy. We saw that there is the attribute PREFECT_ORION_SERVICES_LATE_RUNS_AFTER_SECONDS, but this is only configurable on the self-hosted cloud. Any idea what we could do?
We saw that there is also a health attribute on queue level. But there we do not have the option for the prefect notifications, right?
12/06/2022, 6:06 PM
Hey @Christian Vogel this setting isn't exposed for cloud currently but we've taken that feedback to our product team since, in theory, this should be relatively simple to expose
12/07/2022, 9:32 AM
Cool thanks @Mason Menges. Do you see any other way to check agents/queues being healthy (and being notified if not) except the way we choose?
@Anna Geller we talked about this at the workshop. feature request to add notification if work queue is unhealthy.