Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
w
William Burdett
09/30/2021, 4:14 PMHey everyone, I have an issue. We're getting this error: "Unexpected error: ClientError(\'An error occurred (ThrottlingException) when calling the DescribeTasks operation (reached max retries: 4): Rate exceeded\')"ImportError: cannot import name 'apply' from 'dask.compatibility' (/usr/local/lib/python3.9/site-packages/dask/compatibility.py)k
Kevin Kho
09/30/2021, 4:16 PMHey @William Burdett, looks like you have a mapped task and AWS is throttling those API calls. On Cloud we have task concurrency limiting for situations like this so that APIs aren’t called too much. I don’t think bumping the dask version would help? Though I’m not sure that is going on with that error.
w
William Burdett
09/30/2021, 4:17 PMThis error is showing up in the cloud actually
k
Kevin Kho
09/30/2021, 4:18 PMTask concurrency is not there by default. You have to tag the tasks and then specify the limit for that task tag, and then the submission of those mapped tasks will be limited.
This is the doc for that if it helps
Something seems off with the Dask installation. Did you use conda install?
w
William Burdett
09/30/2021, 4:23 PMWe use pip
k
Kevin Kho
09/30/2021, 4:26 PMThis is not quite your issue, but maybe upgrading
distributedw
William Burdett
09/30/2021, 4:27 PMSo, I added a concurrency limit. Is 20 too high?
k
Kevin Kho
09/30/2021, 4:28 PMI honestly wouldn’t know the AWS limits for that operation. Let me try looking
w
William Burdett
09/30/2021, 4:29 PMA lot of our mapping is firing a ton of queries at a snowflake warehouse. So it can handle quite a bit at once
k
Kevin Kho
09/30/2021, 4:30 PMOh sorry are you using ECS? or is this from creating a Fargate cluster?
w
William Burdett
09/30/2021, 4:31 PMFargate
k
Kevin Kho
09/30/2021, 4:32 PMAm going through this issue
I guess the only potential thing to do is to use the adapt kwargs to scale? But it will be slower.
w
William Burdett
09/30/2021, 5:01 PMOkay, thank you. I'll reduce that and see where it gets us