Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, guys. Has anyone had any experience on running about 39k flows/day (~100 tasks each on average ~ 4 million tasks daily), and how the database and server scales with this kind of volume?

Hi Luis ,

What is your use case , and are you asking with regards to prefect server or prefect cloud ?
That scale amounts to almost 50 api interactions every single second, and that's JUST for task execution.
Each flow and task can do a multitude of requests as it transitions through states 

Hi! It would be prefect server self-hosted

Actually it would be for executing several RPAs

I think that would be significantly pushing the envelope of what prefect server could do assuming it could handle that - you would need to provision your own DB that would be able to handle that throughput, and self-hosted your only option would be vertically scaling the DB to meet resourcing

I would be very surprised if we have anyone in this community at that scale that is running prefect server