I built the first iteration of our data pipeline on luigi several years ago. In addition to @An Hoang’s note about how it handles state, it has fewer features than Prefect or Airflow (e.g. no built-in scheduler, UI is for barebones monitoring, etc.) and the API makes it harder to generate dynamic pipelines. However I could see a use case if you just need “`make` for data.”
In other words, if you just need to ensure that the final step in a pipeline is complete for a given set of inputs without rerunning dependencies that have already run successfully, you don’t have complex branching logic, you don’t need a scheduler (or prefer to use something like cron), and you don’t need to execute pipelines from a web UI, luigi might be worth exploring. I’ve certainly found it easier to setup and deploy than both Airflow and Prefect.
With that said, I think most people find that once they’re at the point of moving their pipelines beyond cron and bash scripts, they’re at the point to benefit from the complexity/feature tradeoff anyway. Personally, I also find the Prefect API a breath of fresh air compared to both luigi (where you define tasks through class inheritance) and airflow.