example on here, the whole flow is taking much longer than expected, only executed at the max rate ~ 2 tasks / second, and the whole flow (100 tasks) takes about 2 minutes. It seems the
rate_limit
function is taking the most time here, and is there any way to reduce the latency?
I was planning to wrap each external API call into a task following the guidance on writing "small tasks", but if I need to use rate limiting then do you recommend batching those requests in each task?
Ji-Oh Yoo
07/03/2024, 1:47 AM
I'm running it on the locally hosted prefect server
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.