Hi, guys. Has anyone had any experience on running about 39k flows/day (~100 tasks each on average ~ 4 million tasks daily), and how the database and server scales with this kind of volume?
09/22/2022, 9:23 PM
Hi Luis ,
What is your use case , and are you asking with regards to prefect server or prefect cloud ?
That scale amounts to almost 50 api interactions every single second, and that's JUST for task execution.
Each flow and task can do a multitude of requests as it transitions through states
09/26/2022, 6:43 PM
Hi! It would be prefect server self-hosted
Actually it would be for executing several RPAs
09/26/2022, 6:49 PM
I think that would be significantly pushing the envelope of what prefect server could do assuming it could handle that - you would need to provision your own DB that would be able to handle that throughput, and self-hosted your only option would be vertically scaling the DB to meet resourcing
I would be very surprised if we have anyone in this community at that scale that is running prefect server