Hello ! I am wondering if there is a way in prefect-dbt to stream the log output of DBT commands to the logs displayed in prefect. We are using both prefect cloud and dbt cloud, triggering dbt jobs via prefect. Please see the attached screenshots. As an ideal state we would like any job triggered via

trigger_dbt_cloud_job_run_and_wait_for_completion

to display the DBT logs in prefect. How would I achieve this?

There isn't anything out of the box with the

prefect-dbt

, but this gist from Jeremy Yeo at dbt Labs could be a good place to start getting dbt Cloud run logs into Prefect

I can't speak to getting the streaming Dbt Job run log specifically but we leverage the prefect_dbt.cloud.jobs.get_dbt_cloud_run_artifact function to get the detailed step information of a Dbt Job run which you don't get from the typical trigger_dbt_cloud_job_run... functions.

run_results_artifact = get_dbt_cloud_run_artifact(
                dbt_cloud_credentials = dbt_creds,
                run_id = RUN_ID_RETURNED_FROM_YOUR_TRIGGER_RUN_FUNCTION,
                path = "run_results.json"
                # step=1
                # The index of the Step in the Run to query for artifacts. The first step in the run has the index 1.
                # If the step parameter is omitted, then this endpoint will return the artifacts compiled for the last step in the run.
            )

I suppose you could throw this in a loop until the job completes where it outputs deltas of run_results_artifact between each loop. There's lots of options available to you using this function depending on your use case. I hope this helps!

That's a great point. If you can wait until the step is completed (I think that's when each json file is created) to get the information, this is a much simpler approach