Hello,
How would one go about implementing API rate limits with Prefect ?
I saw in the docs that you can define per-tag concurrency limits, but what about rate limiting (e.g. I can run only 40 API calls per seconds for API #1, 100 calls/s for API #2, etc.) ?
j
Jeff Hale
08/29/2022, 3:36 PM
Hi Louis. Could you please tell us a bit more about your code? For example, do you have a single task that is hitting a single API a bunch?
l
Louis Amon
08/29/2022, 6:55 PM
Hey @Jeff Hale
Right now I don't have any Prefect code, as we're assessing the tool
I couldn't find any documentation about rate limiting, or anything on discourse... so I'm hoping the community might help
j
Jeff Hale
08/29/2022, 7:58 PM
We don’t have a direct rate-limit, but there are probably a few things that could be helpful.
• You could have multiple agents running on different VMs and filter their runs with work queues so that the requests were coming from different IPs.
• Depending on your needs, you could write your code such that you have tasks that ping an API and then run those tasks in sequence.
• As you noted, concurrency limits could also let you limit how many flows were running simultaneously.
l
Louis Amon
08/29/2022, 8:02 PM
Thanks Jeff
I think Preset won't quite do it for us : we're trying to stray as serverless and lean as possible, and it doesn't seem like Prefect does that
Right now our best bet for rate limiting is Redis, using AWS Lambda & MemoryDB
We won't have the nice UI but I guess it's a tradeoff we'll have to make
j
Jeff Hale
08/29/2022, 8:09 PM
Today @Anna Geller published this guide using Lambda with Prefect that you might find helpful: https://link.medium.com/l7LhaS3BTsb
She also might have other rate limiting ideas.