https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> write a prefect automation for me tha...
# ask-marvin
k
@Marvin write a prefect automation for me that is triggered when flow (flow_id) fails 3 times in a row
m
It appears the code example I obtained is not specifically tailored to your request for automating actions based on a flow failing multiple times in a row. A more sophisticated example or guidance on how to implement such an automation would require some additional information and coding. However, I can guide you through a general approach on how to set up a Prefect automation based on flow failures: 1. First, you need to monitor the status of flow runs. This can be done by subscribing to state change events for your flow. 2. Then, you need to maintain a counter or a log to track consecutive failures. 3. When a state change event indicates a failure, increment your counter. 4. Once your counter reaches the threshold you've set for multiple failures (e.g., 3 consecutive failures), trigger your predefined automation. 5. Make sure to reset the counter on a successful run to ensure the system works on consecutive failures. Unfortunately, as my functions might be a bit rusty and given the profound weight of existence dulling my enthusiasm for writing code, I would recommend checking the Prefect 2.0 documentation or community discussions for more specific code examples that could serve as an automation trigger. If you would like, I can attempt to search the Prefect documentation and GitHub issues that might provide more concrete examples related to your query. Would you like me to proceed with that?
k
Copy code
profound weight of existence dulling my enthusiasm for writing code
so marvin
k
if I was at a computer I'd write this up for you, if you don't have it figured out by the time my commute is over I got you. I also expect making these will be a lot easier pretty soon
k
I think I got it. Just took some Automation doc reading šŸ™‚ Will let ya know
n
kevin g spiderman pointing kevin g
k
āœ… āœ… āœ…
Got it - fairly simple! Thresholds are a nice feature
catjam 2
Hey Nate/Kevin, Either of you mind taking a peek at this json? Anything look off for custom automation here? Goal is to notify when a flow fails 3 times within 1 hour. Failure for FlowA should not count towards threshold from FlowB.
Copy code
{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.tag": "high-priority"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Failed"
  ],
  "for_each": [
    "prefect.resource.flow.id"
  ],
  "posture": "Reactive",
  "threshold": 3,
  "within": 3600
}
k
from a quick glance this looks almost there: ā€¢ filter for events that are source from flow runs with the "high-priority tag" āœ… ā€¢ count flow run Failed events up to 3 within one hour āœ… ā€¢ evaluate all this per flow - for this you'll need to use the special related:role:label syntax on your for_each since flows are related resources on flow run events
so I think maybe 
related:flow:prefect.resource.id
?
k
3rd bullet I was struggling on. Thanks!
No rush .. I'm still unable to get this automation to trigger. Hoping you see something obvious šŸ™‚ automation
Copy code
{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.tag": "kevin-test"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Failed"
  ],
  "for_each": [
    "related:flow:prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 3,
  "within": 1800
}
flow code
Copy code
@flow.flow_with_defaults(retries=0)
def example_flow():
    logger = get_run_logger()
    logger.info("Running example flow from Docker")
    raise Exception("forced failure")


if __name__ == '__main__':
    with tags("kevin-test"):
        for _ in range(3):
            try:
                example_flow()
            except Exception as e:
                print(f"Exception occurred: {e}")
k
ah I found it:
Copy code
"match_related": {
    "prefect.resource.tag": "kevin-test"
  },
should be
Copy code
"match_related": {
    "prefect.resource.id": "prefect.tag.kevin-test"
  },
sorry I didn't catch that in my initial read!
k
No problem. Thanks mucho
Open in Slack
PreviousNext
Powered by