hi all. I have been using spotify's luigi to build workflows with around 15,000 tasks. I am hitting high CPU and memory usage in luigi and am looking around for alternatives. Would prefect work at the scale of 15k tasks?

Hi eamonn and welcome! Prefect is definitely designed with scale in mind, and we have many users on Prefect Cloud running tens of thousands of tasks with ease. A few notes / caveats: • Prefect Server likely could not handle this scale, but Cloud could • if you run your Flow on Dask, you can scale out and track memory usage through the Dask dashboard • if your tasks are inherently high CPU / high memory, Prefect won’t be able to mitigate that as it does not natively manage memory