Also, I would be interested in seeing a comparison of Dagster and Prefect/Orion. Anyone aware of such a comparison.
02/17/2022, 3:41 PM
I'm not aware of any such comparison but to still give you some answer, I'd say that both products have a bit different focus. Prefect (and Orion) is primarily a workflow orchestration tool focused on eliminating negative engineering (i.e. wasted time). Prefect makes it easy to turn any Python script into a workflow and saves your valuable engineering time thanks to features such as retries, caching, results, subflows, passing data between tasks, managing execution states in an intelligent way (e.g., via automations, triggers and state handlers), and many more.
Afaik, Dagster's direction is different as it focuses more on data assets (i.e., what your tasks are producing) than orchestrating tasks and flows and managing the execution states. I think the only way to get an unbiased answer here is to give both products a try and decide which API and architecture fit your use case better.
02/17/2022, 4:00 PM
I did give Dagster a try. Though it had a nice UI it felt a bit "heavy", where even seemingly simple things like running a task multiple (but variable) times took quite a bit of code wrestling. The end result was code that did not feel natural. Also defining data sources as resources felt like a bother.
OTOH Dagster's and Prefect's pith (workflows are just plain python code) seemed similar to me. Dagster is also transitioning to a gen 2 version. The trouble for a beginner is finding relevant non outdated tutorials and videos.
02/17/2022, 4:05 PM
I definitely agree with that sentiment. When it comes to finding the right tutorials, we've recently launched Discourse as a knowledge base with a highly granular categorization through tags to make it really easy for our users to find the right information.