07/17/2019, 6:29 PM
Hi @Akshay Verma, our recommendation for any comparison is to try them both and use the one that you think is best! However, comparison questions are common so I’ll try to give a complete answer here: Prefect and Dagster are both positioned as tools for building data applications. Prefect was designed to solve the negative engineering problem (, and was inspired by hundreds of real-world lessons learned building Apache Airflow. It was explicitly designed to solve those issues. Dagster has so far failed to articulate a specific problem it solves other than claiming “data is broken.” It provides a DSL for defining DAGs over other tools, like Airflow. Frankly, we are as curious as you to see whom that appeals to. Prefect has always had a philosophy that “Python is the API” and any Python code can be transparently transformed into a Prefect workflow. Dagster started with a very cumbersome, explicit DSL, but has more recently adopted some of Prefect’s ideas, like a functional API, or running on Dask. We’re happy to drive innovation, but bolt-on features are often poor substitutes for well-designed ones. From an integration standpoint, Prefect was designed for every single piece to be completely pluggable via a simple API. This means, for example, that our Dask integration can take advantage of Dask’s data serialization features and resource affinities. In contrast, Dagster appears to have taken a “least common denominator” approach to integrations. This means that while Dagster DAGs can be executed on other engines like Airflow or Dask, they don’t take advantage of any of the unique features of those systems. Lastly, our Prefect Cloud platform forms the backbone of the Prefect ecosystem. It has free and paid tiers to support all use cases. We do not see anything comparable in Dagster; perhaps they will introduce something once we make Cloud available publicly.