I’ve set up Automations in my workspace for when work queues enters unhealthy or healthy, and I’ve noticed a lot of false alarms. We have 1 agent running prefect 2.7.9 servicing 2 work queues. We get about 5-6 notifications for unhealthy work queues a day but every time I check they are healthy. Some behavior I’ve observed:
• A work queue unhealthy notification comes in, followed by a healthy work queue notification around a minute later
• A work queue healthy notification comes in with no corresponding unhealthy work queue notification proceeding it
• Only one of the two work queues get unhealthy notification, even though they are serviced by the same agent.
I don’t see any error messages in the logs for the prefect2 agents. Any ideas how to diagnose or reduce the false alarms?