Amanda Wee
07/14/2022, 2:46 AMAnna Geller
07/14/2022, 10:34 AMAmanda Wee
07/14/2022, 12:55 PMAnna Geller
07/14/2022, 1:08 PMuse Airflow DAG factory to allow us to write YAML configs to represent the DAGs with dbt tasks. To me, this negates any advantage that Prefect would provide.this is also possible (and much easier!) in Prefect 2.0 - here is an example https://prefecthq.github.io/prefect-openmetadata/run_ingestion_flow/ But what I meant more is: what if you end up having one edge case where you use more than dbt and this YAML DAG factory is not sufficient? I understand the ease of use of config files, but if I had to choose would pick a tool that is equipped to handle more use cases in the future - you can add constraints only when constraints are needed, rather than choosing a tool that is constrained by default and doesn't allow to move past those constraints when some use case requires that -- this is how I would communicate it with your team - showing that you care about choosing the right tool that will work well long-term rather than one that will work just for now
Kevin Kho
07/14/2022, 4:02 PM