https://prefect.io logo
Title
m

Marius Vollmer

02/27/2023, 4:15 PM
Hello all, I would like to ask for your recommendations on structuring flows/subflows/tasks. We are working with dbt and Prefect and would like to define as many steps as possible as tasks, so that the radar displays the entire job as meaningfully as possible (without having to jump from flow to flow or subflow; instead, we want to see an overall picture). We run into the problem that dbt_cli() itself is also a task (correct me if I'm wrong). What are your recommendations here? How do you proceed with the interaction of Prefect and dbt? Many thanks in advance @Florian Pape, fyi
r

Rob Freedy

02/27/2023, 10:35 PM
Hey @Maryam Veisi!! I have found this blog to be super informative on integrating Prefect and DBT. I would also recommend checking out the prefect-dbt collection (see here). The way you lay out tasks, subflows, and flows is dependent on what you are trying to do with your flow. You could have a single flow with a bunch of tasks that call out to DBT cloud if you would like to have visibility on one page. Let me know if this answers your questions or not, happy to discuss your use case more
m

Marius Vollmer

02/28/2023, 10:33 AM
Hi @Rob Freedy, many thanks for your reply! Any input is gratefully received! 🙂 Against the background of our goal to get as meaningful a radar as possible (as here https://images.ctfassets.net/gm98wzqotmnx/7iONcHLhVeo0TK5zQizWTc/ddeeeab2e5a9498e4f852aa5f1063751/radar_running.gif?fm=webp): How do you structure your flows and tasks? We would like to avoid having to click through a new radar for every dbt cli command. Was/Is this also an issue for you? Or to put it another way: Do you know how I can map the following scenario with dbt and Prefect: Two dbt jobs in level 1, both of which must be completed before job 3 on level 2 can start. We would like to map everything as tasks (instead of subflows). Unfortunately, I am not getting anywhere with the combination of the dbt_cli command with wait_for or similar. Can you help me out here?