eamonn faherty

01/21/2021, 11:11 AM
hi all. I have been using spotify's luigi to build workflows with around 15,000 tasks. I am hitting high CPU and memory usage in luigi and am looking around for alternatives. Would prefect work at the scale of 15k tasks?

Chris White

01/21/2021, 3:55 PM
Hi eamonn and welcome! Prefect is definitely designed with scale in mind, and we have many users on Prefect Cloud running tens of thousands of tasks with ease. A few notes / caveats: • Prefect Server likely could not handle this scale, but Cloud could • if you run your Flow on Dask, you can scale out and track memory usage through the Dask dashboard • if your tasks are inherently high CPU / high memory, Prefect won’t be able to mitigate that as it does not natively manage memory