Dear Prefect community, can someone provide a direct comparison of Prefect and Luigi (https://github.com/spotify/luigi)? I have the impression that Prefect is superior in many aspects, but would like to be sure about that without spending much more time on trying out Luigi myself.

From a reply I saved a while back from Prefect:

Luigi uses a highly-stateful notion of “targets” to determine whether a task finished successfully. These targets conflate data outputs and state outputs, meaning you can’t adjust the state of a task without modifying the data it produced. Conversely, Airflow only supports state outputs. Prefect supports both state and data outputs independently.

I built the first iteration of our data pipeline on luigi several years ago. In addition to @An Hoang’s note about how it handles state, it has fewer features than Prefect or Airflow (e.g. no built-in scheduler, UI is for barebones monitoring, etc.) and the API makes it harder to generate dynamic pipelines. However I could see a use case if you just need “`make` for data.” In other words, if you just need to ensure that the final step in a pipeline is complete for a given set of inputs without rerunning dependencies that have already run successfully, you don’t have complex branching logic, you don’t need a scheduler (or prefer to use something like cron), and you don’t need to execute pipelines from a web UI, luigi might be worth exploring. I’ve certainly found it easier to setup and deploy than both Airflow and Prefect. With that said, I think most people find that once they’re at the point of moving their pipelines beyond cron and bash scripts, they’re at the point to benefit from the complexity/feature tradeoff anyway. Personally, I also find the Prefect API a breath of fresh air compared to both luigi (where you define tasks through class inheritance) and airflow.

Thanks a lot for your answers!

Each of those comments match our experience with Luigi here as well.