https://prefect.io logo
Title
n

Nikhil Jain

02/07/2023, 6:15 PM
Hi, I am still getting
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
Here’s my setup:
prefect==2.7.0
,
prefect-aws==2.1.0
. ECS block definition:
ecs_task_block = ECSTask(
    task_definition_arn=task_def_arn,
    cluster=f'{env}-prefect-cluster',
    image=f'123456789.dkr.ecr.{regions[env]}.<http://amazonaws.com/{image_region[env]}-stride-enrollment-flows-ecr:{ecr_image_tag}|amazonaws.com/{image_region[env]}-stride-enrollment-flows-ecr:{ecr_image_tag}>',
    vpc_id=vpcs[env],
    task_customizations=[
        {
            "op": "add",
            "path": "/networkConfiguration",
            "value": {
                "awsvpcConfiguration": {
                    "subnets" : subnets[env],
                    "securityGroups" : security_groups[env],
                    "assignPublicIp" : "DISABLED"
                }
            }
        }
    ],
    configure_cloudwatch_logs=True,
    stream_output=True,
)
ecs_task_block.save(f'{env}-ecs-block', overwrite=True)
I can see that prefect is still creating new task definitions; one of the things it is overriding on the task def is
aws-logs-prefix
to be the flow-run-id. Is there a different way to be able to access logs in cloudwatch without setting a different name?
I can see couple of ways of fixing this or at least mitigating this issue: 1. set
configure_cloudwatch_logs
to
False
. I am wondering if this will effectively stop logging flow run logs to cloudwatch? Or will they still be available but with a different name? 2. I can try to use different families for each flow so that the request throttling applies separately for different runs instead of collectively. However, I am wondering if this line of thinking is correct because AWS Request Throttling document seems to suggest that the throttling is per account per region. So having multiple families may not help?