I'm seeing all my flows fail with `requests.exceptions.RetryError: HTTPSConnectionPool(host='<http:/...

Brett Naul

06/29/2021, 6:24 PM

I'm seeing all my flows fail with

requests.exceptions.RetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by ResponseError('too many 429 error responses'))

, maybe related to the above error?

Kevin Kho

06/29/2021, 6:31 PM

Hey @Brett Naul, this is related to rate limiting. Are you running a lot of flows at the same time or do you have a lot of running flows? Will ask the team about this

Brett Naul

06/29/2021, 6:32 PM

I didn't think so but it looks like there is kind of a lot, I'll look into why there's so many going

Brett Naul

06/29/2021, 6:50 PM

@Kevin Kho can you give me some hints as to what kind of thing is getting rate limited? is it related to the # of active task runs, flow runs, ...?

Kevin Kho

06/29/2021, 6:52 PM

API calls. Each task state change is an API call. We’ll adjust the rate a bit on our end.

Brett Naul

07/19/2021, 10:04 PM

@Kevin Kho we're seeing this error again so all of our flow runs are stuck. is there somewhere we can see this quota so this doesn't just happen out of the blue?

Brett Naul

07/19/2021, 10:27 PM

^ this specific issue appears to have worked itself out but it would be nice to have some guidance around how to avoid hitting this in the future (I think we were locked out for a couple of hours)

Kevin Kho

07/20/2021, 3:56 PM

It’s my understanding that this got resolved via DM but let us know if you have any followups

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.