Hi all. I am seeing many of our jobs fail with this failure mode
Copy code
requests.exceptions.RetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by ResponseError('too many 429 error responses'))
Ours jobs are quite parallel, and wonder if there is anything that must be resolved, either user or server side that can enable this parallel execution.
k
Kevin Kho
08/23/2021, 10:02 PM
Hey @Vincent, this is related to rate limiting when hitting the Prefect API (updates of task states). Will ask the team about it.
v
Vincent
08/23/2021, 10:04 PM
Thanks!
k
Kevin Kho
08/23/2021, 10:37 PM
What Prefect version are you on?
v
Vincent
08/23/2021, 10:48 PM
We use 0.14.17 and 0.14.22 .
k
Kevin Kho
08/23/2021, 10:49 PM
0.15.4 has the most robust logic for rate limits. Is upgrading to try an option?
v
Vincent
08/23/2021, 10:51 PM
Yes. We can push for that internally. Thanks for the suggestion
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.