```Hey everyone, I have an issue. We're getting this error: "Unexpected error: ClientError(\'An erro...

William Burdett

09/30/2021, 4:14 PM

Copy code

Hey everyone, I have an issue. We're getting this error: "Unexpected error: ClientError(\'An error occurred (ThrottlingException) when calling the DescribeTasks operation (reached max retries: 4): Rate exceeded\')"

I tried to bump the version of dask to 2021.9.1 and it is giving me back this error:

Copy code

ImportError: cannot import name 'apply' from 'dask.compatibility' (/usr/local/lib/python3.9/site-packages/dask/compatibility.py)

I have no idea how to handle this

Kevin Kho

09/30/2021, 4:16 PM

Hey @William Burdett, looks like you have a mapped task and AWS is throttling those API calls. On Cloud we have task concurrency limiting for situations like this so that APIs aren’t called too much. I don’t think bumping the dask version would help? Though I’m not sure that is going on with that error.

William Burdett

09/30/2021, 4:17 PM

This error is showing up in the cloud actually

Kevin Kho

09/30/2021, 4:18 PM

Task concurrency is not there by default. You have to tag the tasks and then specify the limit for that task tag, and then the submission of those mapped tasks will be limited.

Kevin Kho

09/30/2021, 4:19 PM

This is the doc for that if it helps

Kevin Kho

09/30/2021, 4:21 PM

Something seems off with the Dask installation. Did you use conda install?

William Burdett

09/30/2021, 4:23 PM

We use pip

Kevin Kho

09/30/2021, 4:26 PM

This is not quite your issue, but maybe upgrading

distributed

might help also?

William Burdett

09/30/2021, 4:27 PM

So, I added a concurrency limit. Is 20 too high?

Kevin Kho

09/30/2021, 4:28 PM

I honestly wouldn’t know the AWS limits for that operation. Let me try looking

William Burdett

09/30/2021, 4:29 PM

A lot of our mapping is firing a ton of queries at a snowflake warehouse. So it can handle quite a bit at once

Kevin Kho

09/30/2021, 4:30 PM

Oh sorry are you using ECS? or is this from creating a Fargate cluster?

William Burdett

09/30/2021, 4:31 PM

Fargate

Kevin Kho

09/30/2021, 4:32 PM

Am going through this issue

Kevin Kho

09/30/2021, 4:36 PM

I guess the only potential thing to do is to use the adapt kwargs to scale? But it will be slower.

William Burdett

09/30/2021, 5:01 PM

Okay, thank you. I'll reduce that and see where it gets us

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.