https://prefect.io logo
j

Joshua Greenhalgh

07/04/2023, 12:08 PM
Hi I have just received a popup saying I have recently hit rate limits - its absolutely impossible that this is the case - how can I see where and when this happened?
1
c

Chris Reuter

07/04/2023, 12:22 PM
Hi Joshua - you can see more details on rate limiting here. When you hit a rate limit you will receive a
429
response, with a
Retry-After
header that will mean your request is retried. On that page you can see the rate limits for the 3 endpoints that we rate limit on: the
flow_runs
,
task_runs
, and
flows
endpoints. When you receive >=10
429
responses in a week it means that you receive that in-UI popup about hitting rate limits. it is possible that you only briefly experienced rate limiting, your flows were retried after hitting a
429
, and no action is needed. if you continue to hit rate limits & are worried they are impacting your operations, you can chat with a product advocate about what you can do to avoid them.
j

Joshua Greenhalgh

07/04/2023, 12:28 PM
cool thanks - I don't think I have been running anything close to 400 flow runs/task runs/flows per min - would be very helpful to see some record of where these rate limits happened?
c

Chris Reuter

07/04/2023, 12:29 PM
For sure, we're definitely planning on adding them to the UI. Rate limits are one way in which we protect our service's stability, and we didn't want to wait for visibility in the UI to enact them.
j

Joshua Greenhalgh

07/04/2023, 12:33 PM
It concerns me because I cannot see anyway I have got even close to those limits - not to mention I haven't ran any flows until today for over a week
so if its triggered for >=10 in a week - the only thing live was the agent on my k8s cluster over this period so has it been spamming API requests?
c

Chris Reuter

07/04/2023, 12:36 PM
Is it possible that you haven't logged in for a week? It could have been a residual popup that you didn't see until now.
You won't be charged or anything like that, it's just to let you know that you have been hitting rate limits so you are aware.
j

Joshua Greenhalgh

07/04/2023, 12:38 PM
I haven't logged in no - so yeah it may be from before I did - but still - I don't understand how my usage could ever have caused 400 requests to those endpoints per min - I have never had more than one flow running at a time
c

Chris Reuter

07/04/2023, 12:43 PM
There are many different types of API requests you can make to each endpoint. For example, you can Read, Delete, Update, Set State, etc. of a task run. It is also possible that you were just doing one thing, but you had a runaway dataset that you didn't know about (i.e. thousands tasks within one flow would be a common one) We can dive into it with you to a) figure out exactly what happened and b) help you avoid them, but to do so we'd need some more info. OK if a product advocate reaches out to you tomorrow to do that?
j

Joshua Greenhalgh

07/04/2023, 12:59 PM
That would be great - I am still in the testing period and if I am triggering rate limits with my current activity it would be very very concerning! I do not have any flows with 1000s of tasks 4 or 5 max... looking at the docs again there is also a limit on logs - 700 per min - now I did turn on debug logs on my agent so maybe that?
I am actually removing all use of tasks completely due to https://github.com/PrefectHQ/prefect/issues/9934
c

Chris Reuter

07/04/2023, 1:12 PM
Ahh yes - it is more likely logs, those can add up very quickly
👍 1