I wrote about the trends in data orchestrations, and of course, prefect is a part of it 🙂 Hope it's helpful, and looking forward to your comments. I'm covering topics such as: • The shift from Data Pipelines to Data Products • Data-aware Pipelines that know about the inner life of a task • Why a declarative approach with higher-level abstractions helps • How abstractions improve reuse code between complex cloud environments • Where do we come from with the evolution of Data Pipeline Orchestration • Open-source data orchestration tools such as Airflow, Prefect, Dagster https://airbyte.com/blog/data-orchestration-trends

HiSimon, thanks for sharing!

Hi Kevin, you are very welcome. I hope it is useful to people.

Awesome article. Thank you @Simon Späti

Excellent article. Now I'm actually quite curious what the difference will be between dagster and prefect 2. Is that something that somewhere in the calendar to write?

Prefect 2.0 has a bunch of things in the pipeline that it will be premature to write for quite sometime 🙂

And in terms of philosophy of the package? What is one of the main reasons for writing prefect and not using dagster?

I think we are more general purpose while they are more data focused. Their website says “data orchestration”. On the other hand, we don’t look at data or code inside tasks, though that may change in the future as we add more features. But for Prefect, it will always be an opt-in thing for users to show their data to us. Dagster is more opinionated about the data world, and if you opt-in to that model, you can get additional features from the orchestrator. Prefect is more about meeting the user where they are

Thanks for that flavour!