Heya! Is anyone using P Prefect with âď¸ Meltano, dbt dbt? As in
⢠Meltano for the EL
⢠dbt for the T
Then Prefect for the orchestration, monitoring and generally to glue the ELT together
I'm curious what would be the preferred, best practice in plugging in Meltano in Prefect?
I.e. container the meltano project and then run the container as a task in Prefect.. or something else?
There seem to not be much of any blogs on the matter đ¤ˇ
a
Anna Geller
01/16/2023, 6:55 PM
I'm wondering the same! Curious if someone from the community figured that out
I find meltano really interesting, especially given they now even support all Airbyte syncs, but Meltano docs are pushing for using Airflow as orchestrator which is incredibly confusing
đ 1
j
jpuris
01/18/2023, 7:40 AM
I got some responses in Meltano community Slack workspace.
So far almost exclusively it is done by containerising the Meltano project and then using Prefect to run that container or a shell task within the container (never tried, might be deprecated).
In case someone is looking the thread in Meltano's Slack WS, here is it.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.