Hi Team, I'm running into ECS permission issues while trying to deploy a flow-- Though I'm specifying ecs cluster on agent start, I'm getting a permission error on resource *. This user only has permissions to the ECS cluster listed. Any thoughts? Thanks.

prefect agent ecs start --cluster my-cluster-arn --launch-type EC2

botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the RegisterTaskDefinition operation: User: arn:aws:iam::**********:user/********* is not authorized to perform: ecs:RegisterTaskDefinition on resource: *

Hi @Renzo Becerra - it looks like the error is giving you the information you need here; whatever user that is doesn't have

ecs:RegisterTaskDefinition

permissions.

@nicholas this user has ecs:RegisterTaskDefinition for the specific ecs cluster I'm specifying on agent start. It does not have permissions to all (*) ecs resources

Hello Renzo! Asking to narrow the scope of possible options: do you use Dask executor in your setup?

I am not using dask

Hi @Renzo Becerra - can you confirm which version of Prefect you're on?

@nicholas 0.14.12

Great, thank you - we're looking into this, I'll let you know as soon as we have an answer!

Hello Renzo! i was able to replicate the error you are having. The trick is to adjust the permission policy you provide to this user. Some actions such as

ecs:RegisterTaskDefinition

can’t be set only on specific cluster. You might have to set conditions like here:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ecs:PutAttributes",
        "ecs:ListAttributes",
        "ecs:UpdateContainerInstancesState",
        "ecs:StartTask",
        "ecs:RegisterContainerInstance",
        "ecs:DeleteAttributes",
        "ecs:DescribeTaskSets",
        "ecs:DeleteCapacityProvider",
        "ecs:SubmitAttachmentStateChanges",
        "ecs:Poll",
        "ecs:UpdateService",
        "ecs:DescribeCapacityProviders",
        "ecs:CreateService",
        "ecs:RunTask",
        "ecs:ListTasks",
        "ecs:StopTask",
        "ecs:DescribeServices",
        "ecs:SubmitContainerStateChange",
        "ecs:DescribeContainerInstances",
        "ecs:DeregisterContainerInstance",
        "ecs:TagResource",
        "ecs:DescribeTasks",
        "ecs:UntagResource",
        "ecs:PutClusterCapacityProviders",
        "ecs:UpdateTaskSet",
        "ecs:SubmitTaskStateChange",
        "ecs:UpdateClusterSettings",
        "ecs:DeleteService",
        "ecs:DeleteCluster",
        "ecs:DeleteTaskSet",
        "ecs:DescribeClusters",
        "ecs:ListTagsForResource",
        "ecs:StartTelemetrySession",
        "ecs:UpdateContainerAgent",
        "ecs:ListContainerInstances",
        "ecs:UpdateServicePrimaryTaskSet"
      ],
      "Resource": [
        "arn:aws:ecs:us-east-1:<account>:cluster/<cluster_name>",
        "arn:aws:ecs:us-east-1:<account>:task-definition/prefect-<flow_name>:*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "ecs:DiscoverPollEndpoint",
        "ecs:PutAccountSettingDefault",
        "ecs:CreateCluster",
        "ecs:DescribeTaskDefinition",
        "ecs:PutAccountSetting",
        "ecs:ListServices",
        "ecs:CreateCapacityProvider",
        "ecs:DeregisterTaskDefinition",
        "ecs:ListAccountSettings",
        "ecs:DeleteAccountSetting",
        "ecs:ListTaskDefinitionFamilies",
        "ecs:RegisterTaskDefinition",
        "ecs:ListTaskDefinitions",
        "ecs:CreateTaskSet",
        "ecs:ListClusters"
      ],
      "Condition": {
        "StringLike": {
          "aws:TagKeys": [
            "prefect:flow-id"
          ]
        }
      }
    }
  ]
}

In short, you need to update the policy, and it should solve the issue. Also, you can find additional information in AWS documentation

@Mariia Kerimova Thank you for the sharing this! Will I need to include additional policies other than what you have sent me above?

This is a list of main permissions you need to provide, but you might need to add more permissions to the policy, but it’s hard to say without looking at your setup

@Mariia Kerimova Can you please send us the necessary policies for the ecs cluster? and the policies required for an ec2 agent

Hello Renzo. Sorry for long reply. This is minimal policy, but also checkout policies for task-role and execution role in our docs.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "ec2:DeleteSecurityGroup",
                "ecs:CreateCluster",
                "ecs:DeleteCluster",
                "ecs:DeregisterTaskDefinition",
                "ecs:DescribeClusters",
                "ecs:DescribeTaskDefinition",
                "ecs:DescribeTasks",
                "ecs:ListAccountSettings",
                "ecs:ListClusters",
                "ecs:ListTaskDefinitions",
                "ecs:RegisterTaskDefinition",
                "ecs:RunTask",
                "ecs:StopTask",
                "iam:AttachRolePolicy",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:DetachRolePolicy",
                "iam:ListAttachedRolePolicies",
                "iam:ListInstanceProfiles",
                "iam:ListRolePolicies",
                "iam:ListRoles",
                "iam:ListRoleTags",
                "iam:TagRole",
                "logs:DescribeLogGroups",
                "logs:GetLogEvents"
            ],
            "Resource": "*"
        }
}