Hey folks, can someone shed some light again on prefect 2.0 cloud rate limiting? I recently accidentially created a lot of task runs, running in the API rate limit. The rate limit worked as expected, so first of all, thanks for that 😄 But, it seems that most task runs crashed. Why is that? I understand, that prefect agents retry rate limited tasks/flows, but why did the task run crash? And the more important question: Is there a way to prevent the crash and gracefully handle such situations? This was logged by task runs (A lot of the task runs logged this message)

Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Client error '429 Too Many Requests' for url '<https://api.prefect.cloud/api/accounts/bd169b15-9cf0-41df-9e46-2233ca3fcfba/workspaces/f507fe51-4c9f-400d-8861-ccfaf33b13e4/task_runs/e006c65d-95ce-4547-9d2f-2017eabbc282/set_state>'
Response: {'detail': 'Orchestration API rate limit reached'}
For more information check: <https://httpstatuses.com/429>

This was the final flow run log

Finished in state Failed('Flow run encountered an exception. UnfinishedRun: Run is in PENDING state, its result is not available.\n')

The api limits are available in the corresponding page: https://docs.prefect.io/2.11.1/cloud/rate-limits/ Maybe some time rescheduling will help there.

Hi Andreas, thanks for sharing this! To my understanding, 429s are retried up to 5 times by default, with a jitter set between each retry. Both the number of retries and the jitter can be configured by setting values for

PREFECT_CLIENT_MAX_RETRIES

and

PREFECT_CLIENT_RETRY_JITTER_FACTOR

, respectively. Both of these can be set in your prefect settings.

Hey Bianca, perfect thank you for the reference to the client max retries - that's what I was looking for. We'll update to a corp plan in the future, but for now we don't need the rate limit extension. The above scenario was purely on mistake and would also have hit the higher tier rate limit (and caused a chain reaction, as also other flows then hit the limit) - so I'm happy that we have a way to define, how many retries we can undertake. retry jitter is also very handy! Thank you very much again.