Hi all, I'm really excited about Prefect, and am also looking at Dagster and Temporal. Where can I learn more about the differences between the three? Here's my initial understanding. Dagster, Prefect, and Temporal are the most promising options. All of these tools have positive reviews from developers, lots of production users, and a ton of product momentum. Temporal is a workflow manager. Demo code here. Several companies that I know are using Temporal and have very positive experiences. It also is a more generalizable tool that we could use for complex workflows such as device onboarding and other complex microservice sagas. However, it requires more customization to get it working, it’s more difficult to code in because all of your functions have to be able to run asynchronously, and it doesn’t have out-of-the-box data pipeline tools. Prefect takes a generalizable workflow manager approach that Temporal takes, with a bit more data pipeline focus. Demo code here. It’s easier to use than Temporal because code can be synchronous, and it's less opinionated than Dagster because it has fewer abstractions such as I/O management, partitioning, and a software development lifecycle. Unlike Dagster and Temporal, it doesn’t persist task execution results. It’s imperative only, with no declarative option. This imperative philosophy extends to the deployment of the jobs. The Prefect API internal state, not the configuration code, is the state of the jobs. TODO figure out how the other frameworks keep track of configuration state. Dagster specializes in data orchestration as opposed to generalizable workflow management, and it’s more opinionated than Prefect and Temporal. Dagster has more data pipeline specific tools, has a declarative approach to data pipelines as well as an option to do it imperatively, and has a nice software development lifecycle.

Hey Shane - this is a great question, and a difficult one to answer quickly but I'll give it a shot. • Dagster: you described it well; it offers a very data warehouse-centric approach, so if your focus is exclusively within the warehouse and you care most about tracking and managing materializations, and if their programming model resonates with you, it's a good tool. • Temporal: I think Temporal most accurately falls in the "durable execution" category - it's very focused on transactional state management across microservices. So if you need absolute guarantees that a thing will happen once and only once (such as a credit card charge managed by a large distributed application), it's a good candidate. But because of that focus, it can be very heavy and unintuitive for most other use cases (such as data pipelines). • Prefect: Prefect is more general purpose, and IMO is probably best thought of as a workflow application framework; Prefect workflows are highly configurable, portable, and interactive, and often are exposed directly to internal stakeholders and users. As you mention, we do have a focus on data, and most people come to us for data use cases, but often they expand to automating other processes. (small nit: task execution results might not be persisted by default but we do recommend configuring persistence as a best practice) Hope that helps!!

Thanks Chris, this is helpful! How does Prefect configure where flow history (not results) is persisted? I'm looking for something like https://docs.dagster.io/deployment/dagster-instance#dagster-storage. I think the Prefect term for this history is the event feed?

Ah no storage and events are totally different - Dagster storage is the database that a user would use to back their OSS offering, and Prefect has similar database options: https://docs.prefect.io/latest/guides/host/?h=sqlite#prefect-database