Hello folks. I've developed some basic integration of prefect with meltano: https://github.com/Broscorp-net/prefect-meltano This one doesn't use shellcommand to wrap meltano commands, but instead could execute tap and target as separate tasks. It only works with recent singer_sdk based taps/targets but it could be adjusted later. Your comments and PRs are welcome 🙂

hi @Mykhail Martsyniuk - thanks for sharing! looks great catjam one question on the wrapper task you wrote (e.g. here): what made you want to use this pattern?

loop.run_in_executor(None, lambda: your_function(*args, **kwargs))

I see the comment above

Async Prefect task wrapper around run_singer_tap.
    Executes the synchronous run_singer_tap in a thread to avoid blocking the event loop.

but I'm asking because if you have a normal task

@task
def foo(): ...

you can use built-in task functionality to send it to a thread pool via

submit

like

foo() # runs in the main thread

foo.submit().result() # runs in a worker thread

wow, that's cool feedback, thanks! We only use async flows so i created it with this in mind, just to have async tasks too. And reason for async tasks is to actually start piping to the destination before tap completes.