Hi All, We are getting this error in Prefect on an inconsistent basis that looks like hte following...

Vikram Thirumalai

10/04/2021, 2:43 PM

Hi All, We are getting this error in Prefect on an inconsistent basis that looks like hte following below:

Copy code

botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeTasks operation (reached max retries: 4): Rate exceeded

We tried updating Dask but we're not super sure how to proceed since we're still getting the error

Will

10/04/2021, 2:44 PM

Without much knowledge of where in Prefect this call would be made, I do know that is [this](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeTasks.html) API action for ECS

Kevin Kho

10/04/2021, 2:45 PM

Hey @Vikram Thirumalai, this is with a FargateCluster right? I have seen this issue that people still encounter. Are you trying to spin up a large cluster?

Vikram Thirumalai

10/04/2021, 2:46 PM

@Kevin Kho yes its with the Fargate cluster, and I don't think we're trying to spin up anything too large as the runs before and the runs after for the same flow and different flows are fine

Kevin Kho

10/04/2021, 2:48 PM

Is the cluster the same configuration as the flows that run? What this issue suggests is that it really just can’t spin up workers and the “rate exceeded” is because it keeps polling

Vikram Thirumalai

10/04/2021, 7:47 PM

It should be the same configuration but i see, is there a way for us to increase a rate/throttle requests

Kevin Kho

10/04/2021, 7:58 PM

I wouldn’t know for sure if this is a quota that can be set through AWS. You’d need to ask them.

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.