Nelson Griffiths03/22/2023, 4:33 PM
. But could I also match on something like
? And is there a list of the potential values for
that can be matched on somewhere? • From Prefect's view is a schedule a specific implementation of a trigger. Like the event is the clock hits 7 am and then flows get started? Or are the treated differently in the backend? • Is there any plan to tie triggers(automations) to deployments? I know they are a general tool and need to be used for things other than deployments (such as notifications). But it seems to me that for event driven systems also allowing them to be defined with deployments as an alternative to a schedule would be a really good thing? Just curious if this is in the plans or if they will always exist separately. Nothing make or break here, just would be convenient to create a deployment with a trigger set up already. • Is Prefect orchestration the right tool for very event driven systems? Or is Prefect's aim less about that and I should be using something else to watch for events and kick off Prefect deployments via the API? I know there are ways to do this currently in Prefect. And it sounds like there will be some improvements to that soon. I am mostly curious if this is a big focus of Prefect moving forward or if its more of a side use case?
Will Raphaelson03/22/2023, 4:46 PM
For a custom trigger, can you match to a deployment? All the examples I see show a match onOn reacting to specific deployments, you should use a custom trigger with match related for that deployment ID. there’s an example in the docs here for exactly that under custom triggers.. But could I also match on something like
prefect.flow-run.*? And is there a list of the potential values for
prefect.deployment.my-deploymentthat can be matched on somewhere?
From Prefect’s view is a schedule a specific implementation of a trigger. Like the event is the clock hits 7 am and then flows get started? Or are the treated differently in the backend?We’re not using the trigger system in scheduling directly, we use a separate service for a few reasons, one of which is that the observability APIs (triggers system included) are for now cloud only, while scheduling is obviously a necessary feature for our open source software. Generally on event-driven workloads in Prefect, including the interface between events and deployments: Events and triggers are still relatively new to Prefect, but tying them with other concepts users are familiar with is definitely part of the roadmap. I have a blog i’d love to write on the 3-6 month timeline with the title of “beyond scheduling” on using Prefect for “very” event driven systems. To get to this place, we will need to expand and harden a few aspects of the system, in particular: • As you mention, first class event-driven interfaces to deployments - You can kick off deployments from events today via automations, but we’ll be exploring ways to improve the ergonomics, potentially allowing users to attach trigger-like definitions to deployments themselves. • The observability roadmap is highly geared toward making Prefect the best place to watch for events in your data stack and react to them. Of particular interest to you, I think, will be the ability to ingest events into Prefect by hitting a webhook, but we’re also exploring integrations with existing message queues such as AWS SQS for users who already manage events there.
Nelson Griffiths03/22/2023, 4:50 PM