Hi @Nat Busa! This is a great place to post this sort of question. Ultimately, I believe this might be more of an apples to oranges comparison that it might seem at first glance, but here’s my own high level guide:
- if you are looking to easily distribute some large computations or are just looking to take advantage of distributed computation on an ad-hoc basis, you should use dask distributed. If you ever want workflow semantics / retries / scheduling / trigger logic / etc. you should consider hooking dask distributed up to a workflow tool (currently Prefect is the only one that actually supports the power of dask)
- if you are looking to version your machine learning model artifacts on AWS, you should probably use Metaflow. While Metaflow shares some of Prefect’s API and design, it is ultimately not a workflow tool but rather a tool for data scientists to self-serve maintaining their ML model builds with specific design for AWS deployments.
- Dagster appears to be a workflow tool, but I’m not as familiar with it. Ultimately I’d love for a comparison to come from our community, but in the meantime by biggest takeaway from their documentation is that we have fundamentally different philosophies. Prefect strives to be light-weight, and focuses on the surfacing of errors and the ability to specify arbitrary dependencies between jobs / tasks with “workflow semantics” (retries, triggers, etc.). Prefect also has a design philosophy of minimal but sane defaults while simultaneously being deeply configurable. Dagster, on the other hand, seems more interested in what your tasks are doing and consequently is much less light-weight in design and appears to require some intricate configuration. Moreover, last I checked Dagster didn’t support retries which makes me think they aren’t as interested in the “workflow” piece but rather in the configuration aspect. I’m not 100% certain what their target use case is, so I don’t have a good rule of thumb for when you might choose Prefect over Dagster (or vice versa) but hopefully a community member will one day enlighten us all!