https://prefect.io
Join Slack
Having an issue with an automation set up to track Work Queue becoming unhealthy. Configuration is i...
# ask-community
j
Having an issue with an automation set up to track Work Queue becoming unhealthy. Configuration is in the thread. I set the threshhold very low to 30 seconds to test the configuration after having missed a legitimate alert when it was set at 10 minutes. After killing the agent manually, the UI showed the work queue as unhealthy, but the automation never triggered 😕
✅ 1
Copy code
{
  "match": {
    "prefect.resource.id": [
      "prefect.work-queue.f0932238-9512-4b6d-a382-dd4bbf696727"
    ]
  },
  "matchRelated": {},
  "forEach": [
    "prefect.resource.id"
  ],
  "after": [
    "prefect.work-queue.unhealthy"
  ],
  "expect": [
    "prefect.work-queue.healthy"
  ],
  "posture": "Proactive",
  "threshold": 0,
  "within": 30
}
w
hmm, this might be some particularities of how we send these work queue health events. The automation there needs to see a work queue unhealthy event and then not see a work queue healthy event within 30 seconds, but we’ll only emit that first unhealthy event if the work queue changes state. So there’s actually an implicit precondition for this firing which is that the work queue has to be healthy first. Can you try to have the work queue be healthy for a minute and then unhealthy?
j
I can try again, but that was already part of the test. We got the work queue healthy again to get things moving before trying the test.
w
hmm okay, ill try to repro on my end as well.
j
Tried again with a work queue definitely starting in a healthy state, and same results
w
Sorry about that Josh. Mind filing a github issue with your automation, account and workspace info here? https://github.com/PrefectHQ/prefect
actually i was able to reproduce, I’ll file it!
🙏 1
j
@Will Raphaelson Did you end up filing an issue? Not seeing anything in the open issues that looks like this
w
ahh hey josh sorry, i filed this on an internal cloud board, let me file on in the open repo so you can sub (as i said before facepalm )
🙏 1
will do that in a few hours after a few meetings and post back here.
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by