Just researching Does anyone have documentation or reference Prefect Community #prefect-dbt

Just researching: - Does anyone have documentation...

Raul Maldonado

01/25/2023, 7:47 PM

Just researching: • Does anyone have documentation or references on maxing out dbt dbt CORE capabilities before migrating to Prefect?

Dylan

01/25/2023, 7:49 PM

Hey Raul! I don't think I understand the question -- can you share a little more context?

Raul Maldonado

01/25/2023, 7:51 PM

Yup! So, I see a team in my company use Astronomer (Airflow) and dbt core. And they exp the following: • There are times where the processing on dbt jobs are segmented to unnecessary projects (fragile?) • Failed jobs block continuation of pipeline Has someone experienced something similar, documented their experience w/code examples, then migrated to Prefect?

Dylan

01/25/2023, 7:54 PM

Got it! Makes sense

Dylan

01/25/2023, 7:57 PM

We don't have a terribly complex dbt runner setup with Prefect (and we're still running Prefect 1) but some things that I did experiment with / have seen other folks do: • Split the dbt run into a "core" run for models that most/all downstream models depend on • Take a list of tags as an input for independent downstream runs for

run

test

snapshot

so that parts of a project can be run in isolation • Depending on which of those runs fail, reverse ETL & similar jobs may or may not be run depending on what you need

Dylan

01/25/2023, 7:57 PM

We've got a starting point 👉 https://github.com/PrefectHQ/prefect-dbt

Dylan

01/25/2023, 7:58 PM

Curious to see what other folks say!

2 Views

Open in Slack

Previous Next