https://prefect.io logo
Powered by
Title
j

Joshua Grant

01/25/2023, 4:33 PM
Currently running into this issue: 
botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
Our workflow is that we are using an 
ECSTask
block with github storage. 
prefect-aws==0.1.8
prefect==2.6.8
This is mainly caused when a mother flow fires off a subflow via 
run_deployment
, but can occur when trying to invoke a standalone flow
b

Ben Muller

01/25/2023, 10:07 PM
this was fixed with the most recent version of 
prefect-aws
:thank-you: 2
off the top of my head there were some breaking changes, so make sure you read the change log before upgrading.
j

Joshua Grant

01/31/2023, 2:17 PM
@Ben Muller currently getting the same issue with 
prefect==2.7.10
prefect-aws==0.2.4
b

Ben Muller

01/31/2023, 6:09 PM
Does your agent have the new versions?
j

Joshua Grant

02/01/2023, 2:18 PM
yes, agent and server were updated to use the docker image 
prefecthq/prefect:2.7.10-python3.9
, logic used within the Dockerfile: 
FROM prefecthq/prefect:2.7.10-python3.9 
RUN pip install prefect-aws==0.2.4
RUN prefect block register -m prefect_aws.ecs
b

Ben Muller

02/01/2023, 7:16 PM
That's very bizarre. You might have to post what your EcsTask looks like. It might be configured wrong.
j

Joshua Grant

02/01/2023, 7:17 PM
ecs_task_fg = ECSTask(
        aws_credentials=aws_credentials_block,
        image=docker_image,
        cpu=4096,
        memory=8192,
        stream_output=True,
        configure_cloudwatch_logs=False,
        cluster=os.getenv('ECS_CLUSTER'),
        execution_role_arn=os.getenv('FLOW_EXECUTION_ROLE'),
        task_role_arn=os.getenv('FLOW_TASK_ROLE'),
        vpc_id=os.getenv('VPC_ID'),
        env=flow_environment,
        task_definition=task_def,
    )
task_def
is read with `pyaml-env`'s 
parse_config
function and is a dictionary that describes sidcars loaded with the task, it also contains logging configuration for the task container as well (we use DataDog to handle logging). 
flow_environment
is a dict of envars. 
docker_image
is a string of the ECR-stored image
b

Ben Muller

02/02/2023, 8:07 AM
Hmmm this is where my help might run to an end. Not exactly sure of the inner workings, @Anna Geller will most likely be able to help though ☺️
a

Anna Geller

02/02/2023, 1:54 PM
does it work without sidecars? not sure I can help with those patterns
j

Joshua Grant

02/02/2023, 2:59 PM
Just tried without the sidecars and receiving the same error.
seems including the 
task_definition=
argument was the issue. With that completely removed (not just the sidecars), we are not seeing this issue.
👍 1
a

Anna Geller

02/03/2023, 2:48 PM
nice, that makes sense!
j

Joshua Grant

02/13/2023, 6:06 PM
Made the change (removed the 
task_definition=
argument), then went on vacation. When I returned, the too-many-revisions error (which we have been calling this) resurfaced while I was gone. Would love to get some insight into why this is happening.
for anyone else getting these errors, our fix was to define the task definition as part of our CodePipeline and then use that task_definition_arn when creating the 
ECSTask
infrastructure block. This method also allows for the loading of sidecars.
🙌 1
:gratitude-thank-you: 1
#prefect-awsJoin Slack
Powered by