Hi
@Christopher - don’t worry, it’s a natural comparison. In fact, I’m actually an Airflow PMC member. Prefect was explicitly designed to solve the issues with using Airflow in a modern PyData stack (or modern stack period, to be honest) — here’s our own take:
https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4
Typically, the difference between Prefect and most other workflow managers is fundamentally philosophical. For example, Prefect’s goal is to be workflow-agnostic and infrastructure-agnostic. That means our focus is on expressing a really powerful but lightweight syntax for defining workflows, and our challenge is making sure that can be applied in extremely general environments. We want Prefect to become the universal “glue” for whatever tools meet your needs. In contrast, a system like Kubeflow is much more opinionated: it’s for deploying machine learning on Kubernetes. If that’s your only purpose, then Kubeflow may have a more immediate impact, since your work already conforms to its assumptions. If that’s not how you already work, then you’re going to struggle with Kubeflow.
At the end of the day, you should use the best tool for the job. If a more opinionated tool meets your needs, by all means use it! But if you’re experimenting, or you like having a local experience that’s identical to your deployed one, or you work in multiple environments or for multiple stakeholders, you may prefer Prefect’s more general Swiss-army-knife approach. We try to make it as easy as possible for you to dive in and find out!