Currently running into this issue: `botocore.errorfactory.ClientException: An error occurred (Client...

Joshua Grant

01/25/2023, 4:33 PM

Currently running into this issue:

botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.

Our workflow is that we are using an

ECSTask

block with github storage.

Copy code

prefect-aws==0.1.8
prefect==2.6.8

This is mainly caused when a mother flow fires off a subflow via

run_deployment

, but can occur when trying to invoke a standalone flow

Ben Muller

01/25/2023, 10:07 PM

this was fixed with the most recent version of

prefect-aws

🙏 2

Ben Muller

01/25/2023, 10:08 PM

off the top of my head there were some breaking changes, so make sure you read the change log before upgrading.

Joshua Grant

01/31/2023, 2:17 PM

@Ben Muller currently getting the same issue with

Copy code

prefect==2.7.10
prefect-aws==0.2.4

Ben Muller

01/31/2023, 6:09 PM

Does your agent have the new versions?

Joshua Grant

02/01/2023, 2:18 PM

yes, agent and server were updated to use the docker image

prefecthq/prefect:2.7.10-python3.9

, logic used within the Dockerfile:

Copy code

FROM prefecthq/prefect:2.7.10-python3.9 
RUN pip install prefect-aws==0.2.4
RUN prefect block register -m prefect_aws.ecs

Ben Muller

02/01/2023, 7:16 PM

That's very bizarre. You might have to post what your EcsTask looks like. It might be configured wrong.

Joshua Grant

02/01/2023, 7:17 PM

Copy code

ecs_task_fg = ECSTask(
        aws_credentials=aws_credentials_block,
        image=docker_image,
        cpu=4096,
        memory=8192,
        stream_output=True,
        configure_cloudwatch_logs=False,
        cluster=os.getenv('ECS_CLUSTER'),
        execution_role_arn=os.getenv('FLOW_EXECUTION_ROLE'),
        task_role_arn=os.getenv('FLOW_TASK_ROLE'),
        vpc_id=os.getenv('VPC_ID'),
        env=flow_environment,
        task_definition=task_def,
    )

Joshua Grant

02/01/2023, 7:23 PM

task_def

is read with `pyaml-env`'s

parse_config

function and is a dictionary that describes sidcars loaded with the task, it also contains logging configuration for the task container as well (we use DataDog to handle logging).

flow_environment

is a dict of envars.

docker_image

is a string of the ECR-stored image

Ben Muller

02/02/2023, 8:07 AM

Hmmm this is where my help might run to an end. Not exactly sure of the inner workings, @Anna Geller will most likely be able to help though ☺️

Anna Geller

02/02/2023, 1:54 PM

does it work without sidecars? not sure I can help with those patterns

Joshua Grant

02/02/2023, 2:59 PM

Just tried without the sidecars and receiving the same error.

Joshua Grant

02/03/2023, 2:03 PM

seems including the

task_definition=

argument was the issue. With that completely removed (not just the sidecars), we are not seeing this issue.

👍 1

Anna Geller

02/03/2023, 2:48 PM

nice, that makes sense!

Joshua Grant

02/13/2023, 6:06 PM

Made the change (removed the

task_definition=

argument), then went on vacation. When I returned, the too-many-revisions error (which we have been calling this) resurfaced while I was gone. Would love to get some insight into why this is happening.

Joshua Grant

02/20/2023, 6:11 PM

for anyone else getting these errors, our fix was to define the task definition as part of our CodePipeline and then use that task_definition_arn when creating the

ECSTask

infrastructure block. This method also allows for the loading of sidecars.

🙌 1

gratitude thank you 1

7 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.