Dear prefect community, I am not sure whether prefect could be the solution for operational data pipelines / workloads triggered by some notification of an arrival of a new file which a microservice needs to manage. The latency requirement are not critical my concern ist data governance, monitoring, and Maintenance in operation The workloads need to check schemas, transform data into a common Schema (xml/xsd) , compare files / tables on record and field level for reporting # deltas records (reconciliation), and kind of operational impact analysis which would need to request another service with the current file as an input. It's quite operational… my concern with coding everything in Java using spring for developing micro service and Kafka for messaging different states is around data governance, cataloging data, and maintenance of the pipelines in operation when we have like hundreds of customers running these pipelines in parallel… what happens when a Pipeline gets broken?! So… reading about prefect made me thinking suggesting your solution instead. I guess airflow might not be an option here

great question! within the next couple of weeks you should see some recipes for event-driven workflows here https://discourse.prefect.io/t/event-driven-workflows-in-prefect-2-0-serverless-workflo[…]ed-based-on-some-action-rather-than-based-on-schedule/1054