Hi everyone, we have been experimenting with the ECSRun and set it up successfully. I was trying to test if the autoscaling feature works so I created a script to submit 100 flow runs in a loop. for the most part most of the runs succeeded however a few of them failed with error:

An error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 2): Rate exceeded

does anyone know why some of the runs failed?

try this https://docs.prefect.io/orchestration/agents/ecs.html#throttling-errors-on-flow-submission

Hi @Anna Geller could you let us know where to add these variables. we tried adding them to our container definition and are still experiencing the same issue.

I think those must be set on the agent, unfortunately, not on the flow run container or ECS Run... I know it's painful because you need to redeploy the agent now 😕

no problem we will give it a try again.

We are currently using an ECS agent, so do we add the variables in the environment underneath the containerDefinitions like so:

"containerDefinitions": [
        {
            "name": "$ECS_SERVICE_NAME",
            "image": "prefecthq/prefect",
            "essential": true,
            "command": ["prefect","agent","ecs","start","--run-task-kwargs","<s3://xxx-test-bucket/david/ecs-config.yaml>"],
            "environment": [
                {
                    "name": "PREFECT__CLOUD__API_KEY",
                    "value": "xxx"
                },
                {
                    "name": "PREFECT__CLOUD__AGENT__LABELS",
                    "value": "['dev']"
                },
                {
                    "name": "PREFECT__CLOUD__AGENT__LEVEL",
                    "value": "INFO"
                },
                {
                    "name": "PREFECT__CLOUD__API",
                    "value": "<https://api.prefect.io>"
                },
                {
                    "name": "AWS_RETRY_MODE",
                    "value": "adaptive"
                },
                {
                    "name": "AWS_MAX_ATTEMPTS",
                    "value": 10
                }
            ],

@Anna Geller just fyi we are still not able to run it with the current config that @David Wang sent. we are in the same team just so you don’t get confused 😅

I see. Can you try that perhaps from the simple CLI command and attach the env vars directly to the start command?

prefect agent ecs start --env AWS_RETRY_MODE=adaptive ...

Also, I wonder what image exactly does this task definition use, maybe you could try specifying it explicitly?

"image": "prefecthq/prefect:1.0.0-python3.8"