Hi, guys. Has anyone had any experience on running about 39k flows/day (~100 tasks each on average ~ 4 million tasks daily), and how the database and server scales with this kind of volume?
c
Christopher Boyd
09/22/2022, 9:23 PM
Hi Luis ,
What is your use case , and are you asking with regards to prefect server or prefect cloud ?
That scale amounts to almost 50 api interactions every single second, and that's JUST for task execution.
Each flow and task can do a multitude of requests as it transitions through states
l
Luis Henrique
09/26/2022, 6:43 PM
Hi! It would be prefect server self-hosted
Luis Henrique
09/26/2022, 6:43 PM
Actually it would be for executing several RPAs
c
Christopher Boyd
09/26/2022, 6:49 PM
I think that would be significantly pushing the envelope of what prefect server could do assuming it could handle that - you would need to provision your own DB that would be able to handle that throughput, and self-hosted your only option would be vertically scaling the DB to meet resourcing
Christopher Boyd
09/26/2022, 6:49 PM
I would be very surprised if we have anyone in this community at that scale that is running prefect server
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.