hi all. I have been using spotify's luigi to build workflows with around 15,000 tasks. I am hitting high CPU and memory usage in luigi and am looking around for alternatives. Would prefect work at the scale of 15k tasks?
01/21/2021, 3:55 PM
Hi eamonn and welcome! Prefect is definitely designed with scale in mind, and we have many users on Prefect Cloud running tens of thousands of tasks with ease. A few notes / caveats:
• Prefect Server likely could not handle this scale, but Cloud could
• if you run your Flow on Dask, you can scale out and track memory usage through the Dask dashboard
• if your tasks are inherently high CPU / high memory, Prefect won’t be able to mitigate that as it does not natively manage memory