https://prefect.io logo
#prefect-community
Title
# prefect-community
v

Vamsi Reddy

03/02/2022, 7:42 PM
Hi everyone, we have been experimenting with the ECSRun and set it up successfully. I was trying to test if the autoscaling feature works so I created a script to submit 100 flow runs in a loop. for the most part most of the runs succeeded however a few of them failed with error:
An error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 2): Rate exceeded
does anyone know why some of the runs failed?
v

Vamsi Reddy

03/03/2022, 3:30 PM
Hi @Anna Geller could you let us know where to add these variables. we tried adding them to our container definition and are still experiencing the same issue.
a

Anna Geller

03/03/2022, 3:36 PM
I think those must be set on the agent, unfortunately, not on the flow run container or ECS Run... I know it's painful because you need to redeploy the agent now πŸ˜•
v

Vamsi Reddy

03/03/2022, 4:03 PM
no problem we will give it a try again.
πŸ‘ 1
d

David Wang

03/03/2022, 4:12 PM
We are currently using an ECS agent, so do we add the variables in the environment underneath the containerDefinitions like so:
Copy code
"containerDefinitions": [
        {
            "name": "$ECS_SERVICE_NAME",
            "image": "prefecthq/prefect",
            "essential": true,
            "command": ["prefect","agent","ecs","start","--run-task-kwargs","<s3://xxx-test-bucket/david/ecs-config.yaml>"],
            "environment": [
                {
                    "name": "PREFECT__CLOUD__API_KEY",
                    "value": "xxx"
                },
                {
                    "name": "PREFECT__CLOUD__AGENT__LABELS",
                    "value": "['dev']"
                },
                {
                    "name": "PREFECT__CLOUD__AGENT__LEVEL",
                    "value": "INFO"
                },
                {
                    "name": "PREFECT__CLOUD__API",
                    "value": "<https://api.prefect.io>"
                },
                {
                    "name": "AWS_RETRY_MODE",
                    "value": "adaptive"
                },
                {
                    "name": "AWS_MAX_ATTEMPTS",
                    "value": 10
                }
            ],
upvote 1
v

Vamsi Reddy

03/03/2022, 4:17 PM
@Anna Geller just fyi we are still not able to run it with the current config that @David Wang sent. we are in the same team just so you don’t get confused πŸ˜…
a

Anna Geller

03/03/2022, 4:23 PM
I see. Can you try that perhaps from the simple CLI command and attach the env vars directly to the start command?
Copy code
prefect agent ecs start --env AWS_RETRY_MODE=adaptive ...
Also, I wonder what image exactly does this task definition use, maybe you could try specifying it explicitly?
Copy code
"image": "prefecthq/prefect:1.0.0-python3.8"
πŸ‘ 2
27 Views