@Marvin I want to use the prefect_dbt package to run DBT core in a prefect flow. I am trying to produce the summary artifact with the following code but am unsuccessful, no artifact is being produced:

result = DbtCoreOperation(
        commands=[
                    "dbt debug",
                    "dbt deps",
                    "dbt seed",
                    f"dbt build {refresh_param}",
                    "dbt docs generate"
                ],
        project_dir=dbt_dir,
        profiles_dir=profiles_dir,
        working_dir=dbt_dir,
        stream_output=True,
        create_summary_artifact = True,
        summary_artifact_key = "dbt-build-task-summary",
        env={
            "GOOGLE_APPLICATION_CREDENTIALS": service_account_path
        }
    ).run()

It seems like you're trying to use the

prefect_dbt

package to create a summary artifact with

DbtCoreOperation

, but it's not working as expected. Here's what I found: 1. Summary Artifact Creation: The

prefect_dbt

package has pre-built tasks that can create artifacts, which include extra information about dbt Core runs. However, specifics about how

create_summary_artifact

should be configured or potential issues with it aren't detailed directly in the documentation I accessed. 2. Artifact Management in Prefect: Prefect 3.x supports creating various types of artifacts, such as progress and markdown artifacts, using functions like

create_progress_artifact()

and

create_markdown_artifact()

. These might offer alternative methods to manually handle artifact creation if the built-in functionality isn't working as expected. 3. Documentation Reference: For more details, you might want to explore the Prefect DBT Integration Docs for any additional configuration or troubleshooting tips. Given your situation, you might want to: - Double-check the configuration of your

DbtCoreOperation

to ensure all paths and environment variables are correct. - Consider manually creating the artifact using Prefect's artifact creation functions if the built-in option continues to fail. If the problem persists, you might want to open an issue in the Prefect GitHub repository or consult their community for more specific guidance.

Hey!

DbtCoreOperation

doesn't support summary artifact creation. You'd have to use something like

trigger_dbt_cli_command

, or

dbt_build_task

Thanks @Sean Williams, I've been digging through the package code and noticed that. I attempted to use

dbt_build_task

but this method does not have the

env

parameter which I need. I'm looking at the

create_summary_markdown

method to run after the

DBTCoreOperation

If I understand that correctly, you currently can't create the artifact after using

DbtCoreOperation

because the tasks that create the artifact use dbt's Python API to invoke dbt, while

DbtCoreOperation

does not. I think you'd have to set the environment variable before using the build task.

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = some_path

Great, thanks for that Sean, you absolutely right, I'm think I was getting slightly confused with env variables and thought I had to have this declared in the shell operation. Setting the environment variable before does the trick.