Hey guys, having trouble with running a flow on FargateTaskEnvironment. Configuration for the Environment is below..

flow.environment = FargateTaskEnvironment(
        launch_type="FARGATE",
        region="eu-west-1",
        cpu="256",
        memory="512",
        networkConfiguration={
            "awsvpcConfiguration": {
                "assignPublicIp": "ENABLED",
                "subnets": ["subnet-X"],
                "securityGroups": ["sg-Y"],
            }
        },
        family="my_flow",
        taskRoleArn="arn:aws:iam::X:role/CommonSuperRole",
        executionRoleArn="arn:aws:iam::X:role/CommonSuperRole",
        containerDefinitions={
            "name": "my-flow",
            "image": "my-flow",
            "command": [],
            "environment": [],
            "essential": True,
        }
    )

I keep getting this error:

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support log driver awslogs

So my questions : • I’m using the same Uber Role for both the taskRoleArn and the executionRoleArn, probably not best practice but should work? • I’ve thrown every possible log and cloudwatch related policy/permission at it that I can think of, but nothing is taking. I can provide a dump of the permissions if need be? Any help massively appreciated, fairly stumped on it. Is there any way to get more debug info out of it?

hi @Darragh We run the Fargate agent for a bunch of production Prefect Flows. Usually task role and execution role are different, with execution role almost always being the standard one named:

ecsTaskExecutionRole

It looks like yours are set to the same role. See this in AWS docs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html

AWS roles and policies are a wild west all unto themselves 🙂 I followed that guide but no joy. One interesting line in it though,

The task execution role is supported by Amazon ECS container agent version 1.16.0 and later

@Darragh could you try these 2 things (should be quick): 1. Confirm that your AWS account has a role for

ecsTaskExecutionRole

2. Update your FargateTaskEnvironment to use the

ecsTaskExecutionRole

for execution role and your own role for task role, like this:

task_role_arn="arn:aws:iam::<your-aws-account-number>:role/<your-aws-iam-role-name>",
    execution_role_arn="arn:aws:iam::<your-aws-account-number>:role/ecsTaskExecutionRole",

Testing now…

@Darragh I'm surprised that you're getting an error complaining about

"log driver awslogs"

-- we specifically add that to our containerDefinitions but you don't have that in your config. Here's ours:

containerDefinitions=[
        {
            "command": [],
            "environment": [
                {"name": "PREFECT__LOGGING__LOG_TO_CLOUD", "value": "true"},
                {"name": "AWS_DEFAULT_REGION", "value": REGION_NAME},
            ],
            "essential": True,
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "blah-blah",
                    "awslogs-region": "our region",
                    "awslogs-stream-prefix": "blah blah",
                },
            },
        }
    ],

@Joe Schmid We might be looking at different definitions here - mine is for the flow definition, and i think yours is for the agent, is that right?

@Darragh that makes sense now. I'd recommend getting a Flow that uses a simple RemoteEnvironment() to work first. (If you haven't already.)

@Joe Schmid Yeah i have a basic one working for remote, but our major use case is on fargate, im going to test the roles tomorrow on a manual fargate task deploy and see if they work at al!

Those roles definitely need to be set for the agent.

Ah, from what i read i thought they were on the flow only! Ok ill test that, thanks joe

No problem, post here with what you find and we'll get you up and running soon!

Thanks joe! So i might not actually need the fargate task envrionmwnt at all, is that right?

So i might not actually need the fargate task envrionmwnt at all, is that right?

That’s correct. There’s certainly no requirement to use it if you’re using the Fargate Agent.

For people who get the

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support log driver awslogs

error message and might be searching for help in the channel later. Make sure that you spelled

taskRoleArn

and

executionRoleArn

in camelCase and not in snake_case. Just spent a good billion hours looking for the source of the problem everywhere else but in my spelling lol

@Darragh did you get around the

Cluster not found

error? I am facing the same

Hi Maikel, we actually moved away from it for a while due to other problems 🙂 Hoping to get back to it soon and fixing up our remaining problems

Thanks Darragh. If anyone here has seen this error before (and fixed it) can you please share the solution ? 🙂

An error occurred (ClusterNotFoundException) when calling the RunTask operation: Cluster not found