tl;dr How scalable is Prefect Server for scheduling concurrent runs of 10s of thousands of flows? I'm evaluating Prefect Server as a solution for managing a data pipeline. How I envisioned this working was using the Prefect framework as a computationally minimal "glue." Prefect would be the central pipe through which the state of the data flows (but not the data itself). It would hand off all computation to microservices via (a)sync requests and its only concern would be controlling how the state is pushed from service to service. Prefect seems to be a great tool for this job; I really like the API and how quickly a developer can get up and running. However, there are a couple of things that I don't yet have total clarity on. This documentation describes Server as a "highly scalable scheduler" but also says that it may start degrading at "~10-20 tasks running concurrently against a typical Server." Does the above degradation occur because the

typical Server

is deployed to a single node via docker-compose with a single agent, instead of being deployed via something like Helm, which horizontally scales scheduled flows across a cluster of agents? How quickly does the size of the PostgreSQL database generally grow, relative to the number of flows run? Is there any reason that regularly deleting old runs would be any more complicated than as suggested here? Are there any case studies with Prefect Server being used to run 10s of thousands of concurrent flows? I see that there used to be a Nomad Agent, are there any helpful resources for running Prefect on a nomad cluster? Thanks for any advice you are able to share. I have been diving into Prefect for only the past couple of days so greatly appreciate any referential pointers here.

Thanks for a detailed and thorough writeup of your use case! Let me try to unpack the answer into multiple steps to give you an equally thorough response. A bit old but still valid response from Chris on the question of scale: "The open-source Prefect Server was not designed for that sort of scale; as described in this new doc, this is one of the reasons people migrate to Prefect Cloud, which is designed for scale and performance."

Thank you for these detailed responses; they are very helpful. I will give some thought to what you've said about the benefits of using Prefect as a dataflow framework instead of just a scheduler framework, while also considering that the "database is the only stateful component for which you need to ensure that it scales properly." fwiw, my infra coworker has said that Hasura scales well horizontally.