https://prefect.io logo
Title
m

Matt Delacour

03/20/2023, 4:28 PM
Is there an integration between Meltano <> Prefect as there is with Airflow? https://docs.meltano.com/guide/orchestration
โœ… 1
r

redsquare

03/20/2023, 4:39 PM
not directly - some just containerise meltano and shell out from prefect e.g https://github.com/radbrt/prefect_flows/blob/main/projects/jottings/meltano_flow/flow.py
๐Ÿ‘ 2
๐Ÿ™Œ 1
n

Nate

03/20/2023, 5:20 PM
in that vein, there's the
prefect-shell
collection you could use to run your meltano commands
m

Matt Delacour

03/20/2023, 6:49 PM
Yes sure I can definitely use shell tasks ๐Ÿ‘
๐Ÿ‘ 1
h

Henning Holgersen

03/20/2023, 9:19 PM
Note that prefect shell doesnโ€™t log logs, so either use the built-in meltano logging feature to log to file, or if you are on k8s, you might already have a log collector.
y

yijie wang

04/10/2023, 1:45 PM
Hello @Henning Holgersen, I would like to try your solution. What is the best way to handle the state so that the tap can run incrementally? Thank you so much for sharing the solution by the way.
h

Henning Holgersen

04/10/2023, 2:32 PM
@yijie wang I use an external postgres database for the state, and set it via an environment variable in Prefect. The documentation on this worked well for me: https://docs.meltano.com/reference/settings#database_uri
y

yijie wang

04/10/2023, 2:39 PM
Got it. I will test it out. Thank you so much @Henning Holgersen!