https://prefect.io logo
t

Tomek Florek

02/21/2022, 2:29 PM
Hey guys 🙂 Got a question on flow scheduling. I’m using the basic IntervalClock for my flows, with all of them being started at pretty much the same time, which worked fine until now. The numbers of flows increased to 30+ and they started stalling, never finishing. It makes sense since we’re querying the same DBs in them and it’s all run on a single EC2. I’d like to adjust the scheduling, so that they’re starting in small groups, few mins apart. My question is - what’s the best way to do that? First thought is CronSchedule, but as the number of flows starts growing into 100's maintaining those individual schedules might be problematic. Is there another way?
1
k

Kevin Kho

02/21/2022, 2:33 PM
Hi @Tomek Florek! Are you on Cloud? Cuz maybe you can just use Flow Run Concurrency limits to handle this so that the other runs gets queued?
a

Anna Geller

02/21/2022, 2:41 PM
I agree with Kevin, concurrency limits is made exactly for that use case to limit and queue the runs. If you are on Server, you can use this workaround to accomplish the same
t

Tomek Florek

02/21/2022, 2:42 PM
Hey Kevin! We are on Cloud, but not the paid version yet, since we’re nowhere near the 20k limit. This is a good reason to consider it though. Do I understand correctly, that even the paid plan would be free for us, if we don’t hit those limits?
a

Anna Geller

02/21/2022, 2:45 PM
Yes, that's correct. Also, with Orion, the concurrency limits feature is fully open-sourced.
🙌 1
5 Views