https://prefect.io logo
a

Andreas Nigg

07/31/2023, 12:20 PM
Hey folks, can someone shed some light again on prefect 2.0 cloud rate limiting? I recently accidentially created a lot of task runs, running in the API rate limit. The rate limit worked as expected, so first of all, thanks for that 😄 But, it seems that most task runs crashed. Why is that? I understand, that prefect agents retry rate limited tasks/flows, but why did the task run crash? And the more important question: Is there a way to prevent the crash and gracefully handle such situations? This was logged by task runs (A lot of the task runs logged this message)
Copy code
Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Client error '429 Too Many Requests' for url '<https://api.prefect.cloud/api/accounts/bd169b15-9cf0-41df-9e46-2233ca3fcfba/workspaces/f507fe51-4c9f-400d-8861-ccfaf33b13e4/task_runs/e006c65d-95ce-4547-9d2f-2017eabbc282/set_state>'
Response: {'detail': 'Orchestration API rate limit reached'}
For more information check: <https://httpstatuses.com/429>
This was the final flow run log
Copy code
Finished in state Failed('Flow run encountered an exception. UnfinishedRun: Run is in PENDING state, its result is not available.\n')
1
i

Ivan

07/31/2023, 12:33 PM
The api limits are available in the corresponding page: https://docs.prefect.io/2.11.1/cloud/rate-limits/ Maybe some time rescheduling will help there.
b

Bianca Hoch

08/01/2023, 5:24 PM
Hi Andreas, thanks for sharing this! To my understanding, 429s are retried up to 5 times by default, with a jitter set between each retry. Both the number of retries and the jitter can be configured by setting values for
PREFECT_CLIENT_MAX_RETRIES
and
PREFECT_CLIENT_RETRY_JITTER_FACTOR
, respectively. Both of these can be set in your prefect settings.
If you're interested in moving up to an organization-tier plan for more lenient rate limits, or have any other questions about rate limiting, I'd be happy to jump on a call with you!
a

Andreas Nigg

08/01/2023, 5:33 PM
Hey Bianca, perfect thank you for the reference to the client max retries - that's what I was looking for. We'll update to a corp plan in the future, but for now we don't need the rate limit extension. The above scenario was purely on mistake and would also have hit the higher tier rate limit (and caused a chain reaction, as also other flows then hit the limit) - so I'm happy that we have a way to define, how many retries we can undertake. retry jitter is also very handy! Thank you very much again.
🚀 1
1