https://prefect.io logo
Title
s

Simon Späti

06/14/2022, 5:41 PM
I wrote about the trends in data orchestrations, and of course, prefect is a part of it 🙂 Hope it's helpful, and looking forward to your comments. I'm covering topics such as: • The shift from Data Pipelines to Data Products • Data-aware Pipelines that know about the inner life of a task • Why a declarative approach with higher-level abstractions helps • How abstractions improve reuse code between complex cloud environments • Where do we come from with the evolution of Data Pipeline Orchestration • Open-source data orchestration tools such as Airflow, Prefect, Dagster https://airbyte.com/blog/data-orchestration-trends
👍 2
🙂 9
k

Kevin Kho

06/14/2022, 6:08 PM
HiSimon, thanks for sharing!
s

Simon Späti

06/14/2022, 6:11 PM
Hi Kevin, you are very welcome. I hope it is useful to people.
k

Khuyen Tran

06/15/2022, 3:01 PM
Awesome article. Thank you @Simon Späti
🙏 1
s

Sander

06/28/2022, 7:50 AM
Excellent article. Now I'm actually quite curious what the difference will be between dagster and prefect 2. Is that something that somewhere in the calendar to write?
k

Kevin Kho

06/28/2022, 7:52 AM
Prefect 2.0 has a bunch of things in the pipeline that it will be premature to write for quite sometime 🙂
🚀 1
s

Sander

06/28/2022, 8:08 AM
And in terms of philosophy of the package? What is one of the main reasons for writing prefect and not using dagster?
k

Kevin Kho

06/28/2022, 8:33 AM
I think we are more general purpose while they are more data focused. Their website says “data orchestration”. On the other hand, we don’t look at data or code inside tasks, though that may change in the future as we add more features. But for Prefect, it will always be an opt-in thing for users to show their data to us. Dagster is more opinionated about the data world, and if you opt-in to that model, you can get additional features from the orchestrator. Prefect is more about meeting the user where they are
👍 1
s

Sander

06/28/2022, 8:51 AM
Thanks for that flavour!