https://prefect.io
Join Slack
Does the DbtShellTask work with the open-lineage dbt? (`dbt-ol run`)
# ask-community
l
Does the DbtShellTask work with the open-lineage dbt? (
dbt-ol run
)
🙌 1
a
I'd say no but to be sure you would need to give it a try cc @alex
a
It probably wouldn’t work out of the box since it looks like there are some env variables that 
dbt-ol
expects that the 
DbtShellTask
does not set. You could subclass 
ShellTask
and set the necessary env variables to run 
dbt-ol
upvote 1
l
Cool - I’ll try that
Update, it did work with 
DbtShellTask
Heres the code sample
Copy code
with Flow("dbt-transformations") as flow:
    dbt = DbtShellTask(
        return_all=True,
        log_stdout=True,
        log_stderr=True,
        profile_name="transformations",
        environment="dev",
        helper_script="cd transformations",
        profiles_dir="",
    )

    dbt_deps = dbt(task_args={"name": "DbtDeps"}, command="dbt deps")

    dbt_doc_generate = dbt(
        task_args={"name": "DbtDocGenerate"},
        command="dbt docs generate",
        upstream_tasks=[dbt_deps],
    )

    dbt(
        task_args={"name": "DbtOpenLineageRun"},
        command="dbt-ol run",
        upstream_tasks=[dbt_doc_generate],
    )
8 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by