https://prefect.io
Join Slack
Hello! the service started deteriorating in Prefect Cloud, for instance, we woke up to 10 crashed fl...
# ask-community
i
Hello! the service started deteriorating in Prefect Cloud, for instance, we woke up to 10 crashed flows because what seems to be a rate limiting rule
Copy code
Failed to generate job configuration: Client error '429 Too Many Requests' for url '<http://orion-internal:4200/api/accounts>...
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429>
a few more also were stuck at the pending state.. is this sort of rate limiting documented somewhere for the Pro Plan? is it something we can control?
j
hey! Could you share the flow run id you saw this message from?
i
hey Jake! one of them is : 06923b22-93e4-7d48-8000-e4c5ab2a7bb2
j
Thank you! Will take a look and get back to you shortly
Hey, so it looks like you hit your rate limit that is part of your pro plan. Those limits are documented here: https://docs.prefect.io/v3/concepts/rate-limits#rate-limits-and-data-retention If you take a look at the rate limits page under your account plan settings page you can see how you're utilizing your limits. It looks like you burst past your limits and some calls began to fail. These requests are generally driven by lots of new orchestration/runs.
i
Thanks Jake for having a look, The only instances where we interact with the API directly is when deploying, a few times per day at most, so we should worry about how many times Prefect calls the API internally?
j
Usage against the limit is calculated across all API calls (both from the client and users hitting the api directly). Most users who hit the limit tend to be trying to orchestrate a large number of flow runs all at the exact same time
i
Thank you! I also missed all the rate limiting docs. I will see how to improve that.
👍 1
7 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by