https://prefect.io
Join Slack
Just researching: - Does anyone have documentation or references on maxing out :dbt: dbt CORE capabi...
# prefect-dbt
r
Just researching: • Does anyone have documentation or references on maxing out dbt dbt CORE capabilities before migrating to Prefect?
d
Hey Raul! I don't think I understand the question -- can you share a little more context?
r
Yup! So, I see a team in my company use Astronomer (Airflow) and dbt core. And they exp the following: • There are times where the processing on dbt jobs are segmented to unnecessary projects (fragile?) • Failed jobs block continuation of pipeline Has someone experienced something similar, documented their experience w/code examples, then migrated to Prefect?
d
Got it! Makes sense
We don't have a terribly complex dbt runner setup with Prefect (and we're still running Prefect 1) but some things that I did experiment with / have seen other folks do: • Split the dbt run into a "core" run for models that most/all downstream models depend on • Take a list of tags as an input for independent downstream runs for 
run
, 
test
, 
snapshot
so that parts of a project can be run in isolation • Depending on which of those runs fail, reverse ETL & similar jobs may or may not be run depending on what you need
We've got a starting point 👉 https://github.com/PrefectHQ/prefect-dbt
Curious to see what other folks say!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by