Hey guys, what's the best way to run singer taps using Prefect? I was thinking of shelling out to either the individual taps / targets OR shelling out to pipelinewise. I'm not sure about 2 things • Am I missing out by shelling out to some CLI when using Prefect compared to using some of the official integrations? • What benefit would I get by mixing pipelinewise and Prefect? Is there a point in doing that or should I directly shell out to singer.io's tools. I'm kind of new to this 😅

I haven't used singer, but if your tools don't have a python interface and need to be triggered via CLI, I'd recommend using prefect-shell. When you run a

ShellOperation

it'll stream its output by default, so the logs created by executing the command will appear as part of the logs that are visible in the Prefect UI

yep, that's what I meant by "shelling out"

Our dbt integration is built directly on top of

prefect-shell

, it's a lot of extra handling of profiles and target definitions to make using dbt more convenient, but ultimately we're just running

ShellOperation

with dbt cli commands

alright, sounds good, thank you so much! Are you familiar with the benefits of using pipelinewise though?

I have never heard of it 😅