Hi everyone, I'm a long-time user and enthusiast of Prefect. I'm currently exploring a more demanding use case and I'm curious about the scalability of Prefect 2 OSS. Specifically, I'm considering deploying it on GKE and need to handle hundreds of flow submissions per second while orchestrating thousands of concurrent running tasks, most of which are sub 5 second flows. Could someone provide insights into whether the open-source Prefect stack is designed to support this level of load on Kubernetes? Thanks in advance! I came across this StackOvereflow thread from 2020, which advices against using open-source offering for large scale, but wanted to check whether anything changed since then :)

hi @Michal Stolarczyk - its certainly possible to handle this scale in OSS just like we do in cloud, it will just require more setup / maintenance / tuning of your infrastructure + potentially require auxiliary services for example, you probably know this already if you've been using open source, you'd definitely want to have a Postgres db instead of sqlite at that scale is there a specific question around load that you're curious about?

I'm also interested in knowing how the open-source version behaves under different loads (thousands of running tasks). I've noticed some open and closed issues without a solution, indicating that Prefect may struggle to handle such a load or has degraded performance when facing such a load. It would be helpful to have "official numbers" or some benchmarks. Also, it would be great to see a case study or blog post outlining the supported number of tasks or flows without degradation; that would be extremely valuable even for development purposes since y'all will have some numbers to beat. A special place over the docs with "tuning" recommendations or configurations that are well-known within the community would be valuable as well, for both technical and marketing purposes.