Is there an integration between Meltano <> Prefect as there is with Airflow? https://docs.meltano.com/guide/orchestration

not directly - some just containerise meltano and shell out from prefect e.g https://github.com/radbrt/prefect_flows/blob/main/projects/jottings/meltano_flow/flow.py

in that vein, there's the

prefect-shell

collection you could use to run your meltano commands

Yes sure I can definitely use shell tasks 👍

Note that prefect shell doesn’t log logs, so either use the built-in meltano logging feature to log to file, or if you are on k8s, you might already have a log collector.

Hello @Henning Holgersen, I would like to try your solution. What is the best way to handle the state so that the tap can run incrementally? Thank you so much for sharing the solution by the way.

@yijie wang I use an external postgres database for the state, and set it via an environment variable in Prefect. The documentation on this worked well for me: https://docs.meltano.com/reference/settings#database_uri

Got it. I will test it out. Thank you so much @Henning Holgersen!