Michael Hoss

06/24/2020, 8:43 AM
Dear Prefect community, can someone provide a direct comparison of Prefect and Luigi ( I have the impression that Prefect is superior in many aspects, but would like to be sure about that without spending much more time on trying out Luigi myself.

An Hoang

06/24/2020, 11:42 AM
From a reply I saved a while back from Prefect:
Luigi uses a highly-stateful notion of “targets” to determine whether a task finished successfully. These targets conflate data outputs and state outputs, meaning you can’t adjust the state of a task without modifying the data it produced. Conversely, Airflow only supports state outputs. Prefect supports both state and data outputs independently.
upvote 1

Dan Ball

06/24/2020, 11:51 AM
I built the first iteration of our data pipeline on luigi several years ago. In addition to @An Hoang’s note about how it handles state, it has fewer features than Prefect or Airflow (e.g. no built-in scheduler, UI is for barebones monitoring, etc.) and the API makes it harder to generate dynamic pipelines. However I could see a use case if you just need “`make` for data.” In other words, if you just need to ensure that the final step in a pipeline is complete for a given set of inputs without rerunning dependencies that have already run successfully, you don’t have complex branching logic, you don’t need a scheduler (or prefer to use something like cron), and you don’t need to execute pipelines from a web UI, luigi might be worth exploring. I’ve certainly found it easier to setup and deploy than both Airflow and Prefect. With that said, I think most people find that once they’re at the point of moving their pipelines beyond cron and bash scripts, they’re at the point to benefit from the complexity/feature tradeoff anyway. Personally, I also find the Prefect API a breath of fresh air compared to both luigi (where you define tasks through class inheritance) and airflow.
upvote 1

Michael Hoss

06/24/2020, 2:39 PM
Thanks a lot for your answers!

Skip Breidbach

06/24/2020, 5:29 PM
Each of those comments match our experience with Luigi here as well.