https://prefect.io logo
a

Alexis Chicoine

08/31/2023, 12:47 AM
I’m trying to get an automation to work, but I just can’t get the trigger right. Maybe someone can guide me? My goal is to have it trigger when there is no completed flow run for a deployment for a period of time. My last attempt was like this:
{
"match": {
"prefect.resource.id": "prefect.flow-run.*"
},
"match_related": {
"prefect.resource.id": "prefect.deployment.8517b52a-165b-4afa-8a31-491d47b3cc50",
"prefect.resource.role": "deployment"
},
"after": [
"prefect.flow-run.Completed", "prefect.flow-run.Cancelled", "perefect.flow-run.Crashed", "prefect.flow-run.Failed"
],
"expect": [
"prefect.flow-run.Completed"
],
"for_each": ["related:deployment:prefect.resource.id"],
"posture": "Proactive",
"threshold": 1,
"within": 300
}
I started without the after and with the for_each as the default prefect.resource.id, but it seemed to trigger more than I was expecting (more than once per period of time set in within). Not sure what the behaviour is supposed to be for a proactive with no after. When does a new window for the within get created? The after here is attempting to set the window for the within to start when a run enters a terminal state. I’m trying to set the trigger to look at runs in the related deployment, but I don’t think the way I wrote it is correct. So far what I got is that match is the event type that the trigger will look for. Match_related is a way to filter the events like the in this case a specific deployment. for_each I still don’t have a good understanding of it. I’d like to fix this use case, but also if you have some examples where for_each is used with different values or a good explanation of how it works that would be appreciated.
I saw this thread that seemed to have some similar ideas. https://prefect-community.slack.com/archives/CM28LL405/p1690395894595699?thread_ts=1690290608.411199&cid=CM28LL405 Would getting rid of the for_each entirely achieve my goal?
w

Will Raphaelson

08/31/2023, 4:23 PM
hmm so are you checking for flows that kicked off and never completed? I think if thats the case you can just change after to prefect.flow-run.Running, and then expect is as it should be, and then in for each i think there’s a typo. it should should just be
Copy code
"for_each": [
    "prefect.resource.id"
  ],
if you truly just want it to fire when, regardless of anything else that happened, you dont see a flow run completed event, you can remove the after key entirely. so something like
Copy code
{
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.id": "prefect.deployment.8517b52a-165b-4afa-8a31-491d47b3cc50",
    "prefect.resource.role": "deployment"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Completed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 300
}
a

Alexis Chicoine

08/31/2023, 5:23 PM
Ok thanks. I’ll do a test with that. Would I be right to expect that I won’t get more than one message per 5 minutes?
w

Will Raphaelson

08/31/2023, 5:24 PM
yes thats correct
a

Alexis Chicoine

08/31/2023, 5:30 PM
For context we use sentry to notify us of flow failures. But sometimes we mess up the deployment so bad that it fails before we can even connect to Sentry. We tried an automation for a failure within 5 seconds, but we’ve seen that in some cases the flow run is reported as having lasted None time so it doesn’t trigger (I guess it doesn’t evaluate as not being less than 5). Another thing that’s happened is that we sometimes forget a flow is paused so obviously it doesn’t have completed runs. This automation would have a within similar to the schedule of the flow most of the time so that we can have a second mechanism that validates that it ran instead of only relying on sentry that tells us that there was an error.
@Will Raphaelson How long should we expect it to take for a new automation to start working? This deployment didn’t have any runs between 1:15 and 2:15 so I was expecting the trigger to happen sooner. It makes it a bit hard to iterate if it takes so long before it works. But if it keeps running that’s not too bad I’ll just adjust the deployment ids and within on this trigger for the different cases.
Is it possible the events at 2:15 are what kick it off 5 minutes after it completed it didn’t trigger but after another 5 minutes it did?
w

Will Raphaelson

08/31/2023, 7:16 PM
hmm. yeah maybe it took 5 minutes to start the cycle but that is a little mysterious. let me ask a colleague who knows this a little more intimately.
a

Alexis Chicoine

08/31/2023, 8:10 PM
Ok yes. Also not sure if there’s a detail that went wrong in the trigger. I get it to trigger every 5 minutes but when there was a completed run it triggered anyway.
It looks like once a second run happened after that I get two notifications every 5 minutes. I’m doing a new test without the for_each part to see what that does .
w

Will Raphaelson

08/31/2023, 9:21 PM
yeah my colleague recommended removing the for each entirely, DM me if you want to pair on it a bit
a

Alexis Chicoine

08/31/2023, 9:37 PM
Looks like it works without the for_each. There was a run around 5:15 and that notification is missing 🙂 I was thinking of trying to setup a single automation that would cover both deployments of a flow with a for each hooked on the deployment id, but since our deployment pipeline runs for each region (there’s one deployment per region) anyway we’d be creating a separate automation for each. Would I be right in understanding that the for_each basically ends up creating an instance of the automation each time it sees the element mention in for_each? So for my case without for_each it just creates one and it starts working immediately. And when I had the for_each as prefect.resource.id it basically created an instance each time there was flow run. I’m expecting if I did for_each : [“relateddeploymentprefect.resource.id”] it would create an instance each time it sees a run with a different deployment id?
w

Will Raphaelson

08/31/2023, 9:40 PM
yes thats correct. with the caveat that if you use
"for_each": ["related:deployment:prefect.resource.id"]
I think you should remove the match on deployment_id, cause thats already filtering down the trigger criteria to only that deployment.
a

Alexis Chicoine

08/31/2023, 9:44 PM
Thanks for your help. The feature is really neat to help us do some quick things without creating flows for them 🙂
w

Will Raphaelson

08/31/2023, 9:45 PM
sorry my last message didn’t send: if you remove it, you’ll get the behavior for new flow runs of each and every deployment. is this okay? otherwise if you only want it to operate on those two deployments, I think you need to create two automations, one for each deployment.
thanks for the feedback. the custom trigger definition can be a little tricky. I want to build out a bunch more recipes in the docs in the next few weeks.
a

Alexis Chicoine

08/31/2023, 9:47 PM
Yes if you have more examples for how each of the parameters works that will be great. That’s probably the biggest barrier to adoption as when you’re doing these I don’t think it’s that much of a hassle to call the api vs a python sdk, but knowing what to put has been challenging 🙂
w

Will Raphaelson

08/31/2023, 9:50 PM
yeah im going to get a squad together to write a bunch of racipes next week and ill shoot them your way. in the interim let me know if you have any more difficulties.
gratitude thank you 1