What are the best practices around creating “filewatcher” or “polling” type tasks? I believe Airflow has something like this called “file_sensor”.
e
emre
09/28/2021, 1:25 PM
Prefect server exposes a GraphQL api, from where you can do a bunch of things, including triggering a flow run with certain parameters.
So you need to create your own 'sensor', which fill simply send a GraphQL request to trigger a flow with the filename as a Parameter.
How your sensor works will depend on where you store your files. Cloud storages like aws s3 can send notifications on file creation, from which you can trigger anything really.
This post describes the same idea:
https://medium.com/the-prefect-blog/event-driven-workflows-with-aws-lambda-2ef9d8cc8f1a
emre
09/28/2021, 1:26 PM
Or, you can just trigger your main flow frequenty to check for new files, and skip every other task if no files are found. 🙈
😅 1
k
Kevin Kho
09/28/2021, 1:43 PM
Hey @Matthew Seligson, Emre is right here we do not have pollers natively, in favor of flows being triggered by events. But you can have a loop inside a task that checks for a condition as well.
m
Matthew Seligson
09/28/2021, 2:01 PM
Thanks so much @Kevin Kho@emre. Makes sense! Super appreciate the response.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.