Hello guys, Wondering if someone has attempted something similar with DBT + Prefect. https://www.astronomer.io/blog/airflow-dbt-1

Hi @M. Siddiqui, we have some articles about Prefect + Dbt and have a lot of people on this stack: • https://www.prefect.io/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/ • https://airbyte.io/recipes/elt-pipeline-prefect-airbyte-dbt • https://medium.com/slateco-blog/prefect-orchestrating-dbt-10f3ca0baea9

@Kevin Kho thanks for the quick reply ! But I'm trying to achieve a step beyond what these articles state. Currently most examples are of Prefect invoking DBT and then logging its output. But I'm interested in letting DBT build its DAG structure and then extracting it onto Prefect's context to allow us to have greater control over each task's dependency and on what to do if things fail. Hence I was wondering if someone from the community has already done this.

I see what you are saying. I have not seen anyone do this, but given the code snippet they had in the article, i think you can do the same thing. There are just two parts. The first is converting the

manfiest.json

into tasks. You can make a function similar to theirs:

def create_task(node, dbt_verb):
    shell = ShellTask(...)  # or use the dbt task
    return shell

and then use it in your Flow:

with Flow(..) as flow:
    dbt_tasks = {}
    for node in data["nodes"].keys():
        if node.split(".")[0] == "model":
             node_test = node.replace("model", "test")
             dbt_tasks[node] = create_task(node, "run")

and then you will have a bunch of dbt tasks, and then you need to set the upstream tasks which will be similar to their code, except you can do:

dbt_tasks[node].set_upstream(dbt_tasks[upstream_node])

Yeah was thinking of doing something similar ... Will try this out over next few weeks. Once I test this out and iron things out, will try to get a package published for others to easily integrate it. Thanks for the feedback !

Hey @M. Siddiqui, my code snippet might help you kickstart the package: https://discourse.getdbt.com/t/how-to-orchestrate-our-dbt-batch-jobs-to-be-resilient-to-single-data-source-failures/3482/2?u=apanasenco PM me when you get something on GitHub cause I'd love to collaborate/contribute

@Aram Panasenco thanks a lot ! will have a take a look at this when I'll kick things off