Hi everyone! Is there a way to stream dbt core command outputs as they run with

trigger_dbt_cli_command

, into the prefect logs in the most recent version of

prefect_dbt

, e.g.

0.5.0

? For example, in the

prefect-dbt==0.3.1

dbt and prefect used to stream the output of dbt command runs made by

trigger_dbt_cli_command

function. However that's not the case in the most recent version.

I don't have a solution but just writing to say my team recently upgraded and we're having the same issue! Being able to stream the logs back, like before, would be useful.

@Oscar Björhn Can't agree more. The only solution I've found with the new version is like this:

from prefect_dbt.cli.commands import DbtCoreOperation

dbt_run = DbtCoreOperation(
            commands=["dbt run"],
            stream_output=True,
            # dbt_cli_profile=dbt_cli_profile_block,
            # overwrite_profiles=True,
            # project_dir=DBT_REPO_LOCAL_PATH,
        )
dbt_run.run()

This is not perfect, e.g. the stream output is not coloured and has redundant text with every message. however the docs clearly doesn't recommend this for "long-lasting operations" , instead it requires using this as context manager but I didn't not get the logs in that way either. God I miss the old version...

Thanks for creating an issue. This looks like a side effect of switching

trigger_dbt_cli_command

to use programmatic invocation of dbt commands in

0.5.0

. Part of that change allowed us to create a summary artifact, with an example in the screenshot here: https://github.com/PrefectHQ/prefect/releases/tag/2.19.1 This is not the same as streaming logs, but it does have some extra information not found in logs, like the compiled code of failed tests for quicker debugging.

The artifact is really useful, and a big improvement for getting an overview once dbt has finished running! But since my dbt job usually takes over an hour to run, it'd be nice to have the ability to see what it's doing or if something unexpected has happened. It's not an urgent thing for us but I expect a lot of people would appreciate (or even expect) such functionality.

Yeah, I agree. That makes a lot of sense