Hi. Airflow and Dagster has "sensors" that can be used to kick off DAGs.
What is the Prefect way of doing this? Run with a high-frequency schedule (e.g. every minute) and short-circuit the flow run using
case
?
z
Zanie
07/06/2021, 6:05 PM
Sensors are a bit trickier in Prefect because of our hybrid model. There are a lot of options here, basically:
• Make it event driven -- trigger a flow run from your external service when the event occurs as discussed in https://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a
• Use retries or pauses as discussed in https://github.com/PrefectHQ/prefect/discussions/4156
• As you've suggested, have a high frequency flow that either exits early or (what I would suggest) creates a new flow run if the condition is met (one flow is your sensor, one is your actual pipeline)
• Have a looping service that watches for events of interest and creates new flow runs
s
Stéphan Taljaard
07/06/2021, 6:35 PM
All of these are valid options. Thanks, I'll investigate them.