@Marvin In prefect cloud, how do a create an automation that will notify a teams channel when the deployment starts with "dev-dw_"?

To create an automation in Prefect Cloud that notifies a Microsoft Teams channel when a deployment starts with a specific prefix like "dev-dw_", you'll typically need to set up some sort of trigger to listen for deployment events and then send a notification to Teams. Here's how you can do it conceptually: 1. Event Trigger in Prefect Cloud: You would need to create an automation in Prefect Cloud to respond to deployment events. This usually involves setting up an automation rule that triggers on specific conditions, such as when a deployment with a name starting with "dev-dw_" begins. 2. Integration with Microsoft Teams: To send a notification to a Teams channel, you'll typically use a webhook. You would create an incoming webhook in the Teams channel where you want to send notifications. Prefect Cloud might not have a direct integration with Teams, so you'd handle the actual messaging via webhook. 3. Connect the Automation to Teams using a Webhook: In your Prefect Cloud automation rule, you'd configure an action to send an HTTP request (the webhook call) to the Teams channel whenever a deployment with the specified prefix starts. Now, I will search the relevant Prefect documentation for more detailed information that corresponds to the Prefect version you're using. Please hold on. It seems I made an error trying to search the documentation. Let's attempt that search again with specific queries. I'll be looking for information related to creating automations in Prefect cloud and integrating with webhooks for notifications. Please hold on. Based on the information from the Prefect documentation for both 2.x and 3.x versions, here's how you can set up an automation to notify a Microsoft Teams channel when a deployment starts with a prefix "dev-dw_": 1. Create an Automation in Prefect Cloud: - Go to the Automations page in your Prefect Cloud account. - Click the

+

icon to create a new automation. - Set a trigger condition that corresponds to the start of a deployment with the name starting with "dev-dw_". You could use deployment event triggers for this purpose. - Define the action you want, such as executing a webhook to your Teams channel. 2. Set Up and Use Webhooks for Notifications: - You’ll need to set up a webhook in Microsoft Teams to receive incoming messages. This can be done by creating an Incoming Webhook in the Teams channel where you want the notifications. - In Prefect Cloud, you can manage webhooks through the UI or via the Prefect CLI. You will create a webhook that sends a notification to the Teams channel whenever the trigger condition is met. - The workflow will involve configuring the webhook with the necessary URL and message template to translate incoming events into Prefect events that trigger your automation. 3. Use Webhooks for Actions: - Configure your Prefect webhook with the Teams webhook URL to enable direct communication. - You might use a simple HTTP POST request to send a JSON payload containing the notification message details to Teams. Here's some helpful documentation I've referred to: - Automations in Prefect 2.x - Webhooks in Prefect 2.x - Creating Automations in Prefect 3.x - Webhooks in Prefect 3.x With this setup, you can monitor your deployments and automatically notify your Teams channel whenever certain deployment events occur. Let me know if you have any more questions or need further help!

hey @Bryan as in, notify when a deployment named like that is created? or when a deployment named like that runs?

Based on the deployment name

sorry, but whats the actual event you want to trigger on? • deployment creation (a deployment got named a certain way) • execution of a flow run that happens to be from a deployment w a certain name

The situation is that we have a current Automation that notifies us when a flow crashed, failed, or timedout where there is a tag of environment:dev. The problem is that this is too broad and we need to add more criteria where the deployment name starts with "dev-dw_" because we have other deployments where the notifications need to go elsewhere

To create a custom trigger for an automation in Prefect Cloud that activates on all flow runs with a specific tag ("environment:dev") and where the deployment name starts with "dev-dw_", you can use a combination of conditions in the Prefect Cloud automation settings to specifically cater to these criteria. Here's how you can conceptualize and set up this custom trigger: 1. Access Prefect Cloud Automations: - Navigate to the Automations section in your Prefect Cloud interface. 2. Create a New Automation: - Click on the

+

icon or option to create a new automation. - Name your automation appropriately, something like "Dev Environment Flow Run Trigger". 3. Define Trigger Conditions: - Set conditions that match your requirements. You will need to define a trigger condition that checks for both: - Tags: Ensure the flow run contains the "environment:dev" tag. - Deployment Name: Set a condition to match deployment names that start with "dev-dw_". 4. Configure the Custom Trigger: - Unfortunately, Prefect's UI might not directly allow a complex condition with an "AND" clause combining tags and name pattern. Therefore, this would typically involve writing a small script or using Prefect's API to programmatically check and trigger actions when these both conditions are satisfied. 5. Custom Trigger Logic (using Prefect's API or your event handler): - If the UI doesn't support this complex logic directly, you may need to resort to a script-based solution using Prefect's API. This can involve: - Listening for all flow run events. - Filtering those events by checking if the deployment name starts with "dev-dw_" and includes the "environment:dev" tag. - Triggering any subsequent action (e.g., a webhook or notification) programmatically. 6. Action on Trigger: - Once your trigger condition is set up, outline one or more actions that should occur when the trigger fires, such as sending notifications, running additional checks, or executing another task/flow. Ensure you have permissions to create automations and read access to tags and deployments when setting up these triggers. You may also need to write a wrapper that continuously polls and checks for these conditions using Prefect’s API if the built-in capabilities fall short. For specifics, you may not find a direct API method to achieve a combined trigger, but you can regularly check Prefect's API documentation for updates or community plugins that might offer enhanced features. Here are where you might focus on for implementing or customizing such triggers: - Automations in Prefect Documentation - Prefect API Documentation If you need more detailed guidance on scripting the automation logic, please let me know!

@Marvin no look for examples with match_related to match on tags and deployment name and expect flow run Completed events

To create a custom automation trigger in Prefect Cloud that matches flow runs with a tag of "environment:dev" and deployment names starting with "dev-dw_", while expecting events when flows complete, you can use the

match_related

feature in combination with trigger conditions and expectations. Here's a general guideline on how you can set this up: 1. Create the Automation in Prefect Cloud: - Go to the Automations section in your Prefect Cloud interface. 2. Define a Custom Trigger with `match_related`: - Use the

match_related

object in your event trigger schema to specify conditions related to your flow runs and deployments. 3. Set Specific Match and Expect Conditions: Here’s how you might define your trigger using Prefect's schema for automations, specifically focusing on your requirements:

json
{
  "type": "event",
  "posture": "Reactive",
  "expect": ["prefect.flow_run.completed"],
  "match_related": {
    "prefect.resource.tags": "environment:dev",
    "prefect.resource.name": "dev-dw_*"
  }
}

4. Configuration Details: - Type: Set to

"event"

since you are triggering based on an event. - Posture: Use

"Reactive"

because you want to trigger on the presence of a completed flow run event. - Expect: Specify

"prefect.flow_run.completed"

to only consider events where flow runs are completed. - Match_related: Use this field for matching tags and deployment names. The

name

field supports patterns such as

"dev-dw_*"

. 5. Action Setup: - After defining the trigger, set up the desired action, such as sending a notification or executing another task/flow. References: - Prefect 2.x Automations Overview - Prefect 3.x Automations Triggers - Using Webhooks and Events in Prefect 3.x Make sure to keep this schema inside the automation settings of your Prefect Cloud. Let me know if you need more specific details on implementing these configurations!

that should be pretty much right

Will do. I am attempting to setup a new automation right now.

I apologize for the oversight. It seems there might be a misinterpretation of how to use tags in the match criteria for Prefect automations. Let me correct that by providing a more accurate approach for using tags in

match_related

conditions. To properly trigger on flow runs with a specific tag and deployment name, the setup should use the proper tagging and naming conventions. Here is how you might adjust your configuration:

json
{
  "type": "event",
  "posture": "Reactive",
  "expect": ["prefect.flow_run.completed"],
  "match_related": {
    "prefect.flow_run.deployment_name": "dev-dw_*"
  },
  "match": {
     "tags": ["environment:dev"]
  }
}

### Explanation: - Type: The type is

"event"

as this automation is responding to events. - Posture: Using

"Reactive"

posture to respond to when flow runs are completed. - Expect: Set to

["prefect.flow_run.completed"]

to only consider events for completed flow runs. - Match_related: This is specifically targeting deployments with names starting with

"dev-dw_"

. - Match: You correctly filter on tags using this section for the

tags

field, which is a direct attribute of the flow run. With this setup, ensure that you correctly have these match fields in your automation configuration. Prefect's event model can indeed be particular with its requirements for event conditions. For ongoing adjustments and specifics, you might want to check the Prefect documentation or forums for direct examples related to tags and conditions. Also, if the Prefect version affects functionality, keep an eye on Prefect's release notes or updates to see if there are changes to their handling of tags in automation. If there's anything else you need, feel free to ask!

@Marvin I want to match on all flow runs where the tag is "environment:dev" and the deployment name is like "dev-dw_".

To set up an automation trigger in Prefect that matches flow runs where the tag is "environment:dev" and the deployment name begins with "dev-dw_", you'll need to configure the

match

and

match_related

fields appropriately within your Prefect automation setup. Here is how you can define such a configuration: 1. Define the Trigger: - Use the

match

property to filter flow runs based on tags. - Use the

match_related

property to filter based on deployment names. Here's a sample configuration that you might use:

json
{
  "type": "event",
  "posture": "Reactive",
  "expect": ["prefect.flow_run.completed"],
  "match": {
    "tags": "environment:dev"
  },
  "match_related": {
    "prefect.resource.deployment_name": "dev-dw_*"
  }
}

### Breakdown: - Type and Posture: This is an event trigger with a posture of "Reactive" to respond to any flow run completion events. - Expect: Set to

"prefect.flow_run.completed"

to filter for completed flow runs. - Match: This part enables filtering based on tags and should aim to match any flow run that contains the

environment:dev

tag. - Match_related: Focuses on related resources, like deployments, and filters names starting with

"dev-dw_"

. ### Execution Context: - Event Triggering: When a flow run completes (trigger event), this configuration will check if it matches your specified tag and deployment name conditions. - Automation Response: When all these conditions are satisfied, the configured action in your automation will execute, such as sending a notification or triggering another flow. You can’t directly specify certain configurations via the Prefect UI if it doesn't allow complex tagging operations, so scripting via Prefect's API might be necessary. For precise implementation details or syntax updates, it's helpful to reference Prefect’s updated documentation: - Prefect 2.x Automations - Prefect 3.x Automations Let me know if you need additional help or if there's anything specific you’d like to explore further!

@Marvin, the below trigger does not work when the "prefect.resource.tags" line is included. How can I fix this?

{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.name": "dev-dw_*",
    "prefect.resource.tags": "environment:dev"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Crashed",
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Failed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}

When configuring triggers in Prefect automations, using

match_related

to filter tags might not work as expected because tags are not directly associated with the deployment resources in the same way that name or identifiers are handled. Here's how you can adjust your setup to fix the issue: ### Suggested Changes: 1. Separate Tag Matching: - Instead of using

match_related

for tags, move the tag matching to the main

match

block since tags are typically associated directly with the flow run itself. 2. Configuration: Use the

match

block to include tags, and reserve

match_related

for filtering related resource traits like deployment names. Here's how you might configure your trigger:

json
{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*",
    "tags": "environment:dev"
  },
  "match_related": {
    "prefect.resource.name": "dev-dw_*"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Crashed",
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Failed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}

### Explanation: - Tags in Match Block: Place

"tags": "environment:dev"

in the main

match

block, which indicates that the tags are to be directly checked on the primary resource (the flow run, in this case). - Deployment Name in Match Related: Keep deployment name checks in

match_related

, as they often pertain to resources related to the flow run. ### Follow-Up: If you continue to experience difficulties, verify whether Prefect's current automation handling or API requires an additional step or modification, and check their API documentation or community forums for updates or examples. If you have more questions or need further assistance on how to implement and test this setup, feel free to ask!

@Marvin, the below trigger below does not work when the "tags" line is included. How can I fix this?

{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*",
    "tags": "environment:dev"
  },
  "match_related": {
    "prefect.resource.name": "dev-dw_*"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Crashed",
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Failed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}

After reviewing the Prefect documentation, it appears that including a tag filter directly within the

match

field as

"tags": "environment:dev"

might not be supported as you attempted. Tags might require a different approach for filtering in Prefect automations. ### Alternative Configuration: To achieve filtering by both deployment name and tags, consider adjusting your strategy: 1. Custom Event Handling: - Use custom events or environment variables to label flows and check those in automation conditions. - Alternatively, leverage Prefect’s Flow Context to pass tag-related information as part of the flow metadata or name. 2. Post-update Filtering: - If you can’t directly use tags in automation, consider handling tag-based filtering in another layer, like via a script or additional Prefect task that completes post-run analysis and manages notifications or logs based on tags. Here's a revised approach you could take if tags need to be included in another part of your workflow:

json
{
  "type": "event",
  "match": {
    "prefect.resource.id": "prefect.flow-run.*"
  },
  "match_related": {
    "prefect.resource.name": "dev-dw_*"
  },
  "after": [],
  "expect": [
    "prefect.flow-run.Crashed",
    "prefect.flow-run.TimedOut",
    "prefect.flow-run.Failed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1,
  "within": 0
}

### Additional Considerations: - Prefect API or CLI: Use the Prefect API or CLI to create additional scripts that handle specific filtering logic and perform necessary operations when these conditions are met. - Wrapper or Middleware: Use a wrapper or middleware that enriches events with extra metadata, making it easier to filter directly by tags post-execution. Documentation Links: - Prefect 2.x Automations Overview - Prefect 3.x Automations and Custom Triggers If you continue to face challenges using tags in automations, checking Prefect’s repository for related issues or community solutions can also be a valuable resource. Let me know if there's anything else specific you would like to delve into!

i think marvin is just doing a poor job here ill try to make an example here shortly

@Nate, I am interested in how this is normally handled by other organizations. I look at automations as more platform configuration. Would this code normally exist in a project/repo by itself which is responsible for configuration? (i.e. similar to an infrastructure as code situation)

so it really depends on how you like to deploy your flows • if you like the

prefect.yaml

cli experience, you can store the triggers as yaml or json like this • if you're using .deploy or .serve a lot, you might prefer the python like I have above, but of course in python you're free to

json.loads

or

yaml.safe_load

any file from disk to make your triggers too

Gotcha. It sounds like to make this work though we would have to add it to all of our projects. Is that accurate?

make this work

by this, do you mean adding triggers to all your deployments?

Yes

yeah so the easiest way in my opinion is to co-locate your trigger definitions with the flows that will use them (together, as a part of a Deployment definition) but if you're interested, you also have the option of creating automations that just happen to refer to your deployments like this so strictly speaking you dont need to co-locate your triggers and flows, but thats the most common pattern

Co-locating the trigger with deployments is going to be a bit of an effort as we have several jobs. Right now our automation is alerting us when there are failures in a given environment. You can think of it as an automation to serve the entire environment. It sounds like by implementing the trigger on the deployment, it becomes an automation per deployment. Since we have 3 environments for every project, our automation count will become 3 times the quantity of projects we have. I will have to talk to my team about this and see how we want to proceed.

Since we have 3 environments for every project, our automation count will become 3 times the quantity of projects we have.

makes sense you may want to consider a single place where you use the

Automation

class directly to create triggers (a special type of automation) for all your environments' deployments

Any chance there is something on the roadmap to make automations a bit easier to create when there are 2 to n pieces of criteria at play?

issues and discussions are the best place to look in that regard, I'd have to look myself feel free to create an issue or discussion if you dont find anything

Will do. Thanks for the help Nate!

👍

@Nate I don't see a channel for issues or discussions. Where were you referring to when you mentioned?

• github discussions • github issues

Ahhh... Thank you!