Hi all! We are using DBT standalone (cli) right now. I’ve been doing some research on an orchestration tool that can handle retries, etc. At first I thought of Jenkins but then airflow was recommended. After even more reading, Prefect seems to one more recommended choice if you are green fielding. I’m having trouble finding further documentation to answer my questions: Does Prefect replace Airflow? Would it replace parts of DBT as well? Thank you!

Hi @Matt Juszczak! Excellent question - Prefect definitely replaces Airflow wholesale and seeks to solve for a lot of the problems that the legacy architecture of Airflow introduced (scalability, high availability, pythonic, extremely testable, etc.) in way that is simple and lightweight yet still deeply configurable. I’ve seen many projects that attempt to alleviate some of these pain points by building DSLs on top of Airflow but ultimately all the fundamental pain points persist. Depending on how familiar you are with Airflow, we have a blog post on the major differences between Prefect and Airflow that you might find interesting: https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 For DBT specifically, we have many users who have integrated DBT with Prefect successfully, so while we might replace parts of DBT (scheduling, visibility, etc.) we probably wouldn’t seek to replace all of DBT. Prefect cares much more about the surface around your tasks (things like dependency structure, capturing and surfacing errors, retries) and much less about what your individual tasks do - this allows for users to orchestrate arbitrary and diverse workflows all using a common tool and API. DBT, on the other hand, cares deeply about what your tasks do and helps you specifically orchestrate SQL queries, so it is much more constrained in scope.

Thanks Chris! That’s really helpful. Is there good integration with the DBT process itself? For example, being able to restart a failure within DBT itself, or executing only part of the DBT pipeline? Right now we are using Singer pre-DBT to extract the data. The pain point we are trying to solve for now is scheduling DBT pipelines to run (not in Cron) and being able to have better control and visuals into specific failures within DBT itself. We have no experience with Airflow, mostly just researching what kind of tool we need. TBH Jenkins seemed to fit the bill for now except for that it won’t provide any “zoomed in” look inside DBT.... but I wasn’t sure if these other tools will either.

It depends on how you choose to integrate the two; as of now I’ve seen users use a Prefect

ShellTask

to call out to the DBT CLI. I’m not as fluent with their CLI, but which commands you break apart into separate Prefect tasks will determine the level of “zooming in” you can achieve. For example, if you use prefect to call out to execute a full DBT run through a single CLI command, then that’s only 1 Prefect Task. However, if you break up the Prefect DAG to mirror the DBT dag then you can gain additional visibility.

Got it! That’s helpful. Final question — I read the post on your differences from airflow. Do you also differentiate yourself from dagster or is that project still too new to write something up?

We haven’t done a deep dive into Dagster as it hasn’t come up as much; however, what I have seen is that workflow features such as scheduling and retries are second-class citizens (I believe their demo examples all run on top of Airflow, for example), whereas in Prefect those are primary value drivers. I think they are aiming for an improved developer experience of some kind, but their abstractions and API always seemed very configuration heavy to me personally. So in that sense, the most noticeable difference is that we have very different design philosophies - Prefect strives to be very minimal (we joke that “when things work well you should forgot Prefect exists”). If you end up learning more about both tools and have any takeaways, both myself and others in the community would love to hear what you find!

Thank you, Chris! I’ve been doing a lot of reading/research and will definitely have some takeaways!

awesome, anytime! let us know if you come up with any other questions and good luck with your team’s project!

Will do! Thanks!