Hi. <Airflow> and <Dagster> has "sensors" that can...
# ask-community
s
Hi. Airflow and Dagster has "sensors" that can be used to kick off DAGs. What is the Prefect way of doing this? Run with a high-frequency schedule (e.g. every minute) and short-circuit the flow run using
case
?
z
Sensors are a bit trickier in Prefect because of our hybrid model. There are a lot of options here, basically: • Make it event driven -- trigger a flow run from your external service when the event occurs as discussed in https://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a • Use retries or pauses as discussed in https://github.com/PrefectHQ/prefect/discussions/4156 • As you've suggested, have a high frequency flow that either exits early or (what I would suggest) creates a new flow run if the condition is met (one flow is your sensor, one is your actual pipeline) • Have a looping service that watches for events of interest and creates new flow runs
s
All of these are valid options. Thanks, I'll investigate them.