Tyler Matteson
05/06/2022, 7:27 PMsubprocess
). What should I know? What am I not asking? Tasks are a mix of polling, ETL on demand and scheduled ETL tasks. I think most data professionals would describe the load as "not much", so efficiency is going to take back seat to maintainable and easy to use.Anna Geller
Tyler Matteson
05/06/2022, 7:36 PMAnna Geller
Tyler Matteson
05/06/2022, 7:42 PMAnna Geller
Tyler Matteson
05/06/2022, 7:43 PMAnna Geller
Tyler Matteson
05/06/2022, 7:51 PMAnna Geller
They have an "orchestrator" abstraction (Airflow and Dagster)This is exactly the part I'm most confused about! How is that supposed to work?! š it would assume that Airflow's or Dagster's logic is translatable to Prefect and vice versa which is not true. Prefect 2.0 is a completely different tool than Airflow or Dagster, it doesn't even require you to build a DAG. I guess, perhaps Meltano assumes that a DAG is a requirement for orchestrator and they can translate a DAG from one orchestrator to another, but this is not doable IMO - I would love to be proven wrong here
Tyler Matteson
05/06/2022, 9:00 PMAnna Geller
Tyler Matteson
05/06/2022, 9:20 PMwithout scheduling and real orchestration backendI would say it delegates orchestration. Airflow is a very real tool but not built in. This is splitting hairs to be sure, but this characterization would matter if you described it to somebody who had deep experience with Meltano and wanted to quibble.
Anna Geller
With qualifiers for my inexperience, yes.yes as EL tool or as orchestrator? š
Tyler Matteson
05/06/2022, 9:25 PMAnna Geller