Title
r

Renzo Becerra

03/19/2021, 6:44 PM
Hi Team, I'm running into ECS permission issues while trying to deploy a flow-- Though I'm specifying ecs cluster on agent start, I'm getting a permission error on resource *. This user only has permissions to the ECS cluster listed. Any thoughts? Thanks.
prefect agent ecs start --cluster my-cluster-arn --launch-type EC2

botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the RegisterTaskDefinition operation: User: arn:aws:iam::**********:user/********* is not authorized to perform: ecs:RegisterTaskDefinition on resource: *
n

nicholas

03/19/2021, 6:48 PM
Hi @Renzo Becerra - it looks like the error is giving you the information you need here; whatever user that is doesn't have
ecs:RegisterTaskDefinition
permissions.
r

Renzo Becerra

03/19/2021, 6:50 PM
@nicholas this user has ecs:RegisterTaskDefinition for the specific ecs cluster I'm specifying on agent start. It does not have permissions to all (*) ecs resources
👀 1
m

Mariia Kerimova

03/19/2021, 7:43 PM
Hello Renzo! Asking to narrow the scope of possible options: do you use Dask executor in your setup?
r

Renzo Becerra

03/19/2021, 7:48 PM
I am not using dask
@Kyle Moon-Wright
@Darren
n

nicholas

03/23/2021, 4:41 PM
Hi @Renzo Becerra - can you confirm which version of Prefect you're on?
r

Renzo Becerra

03/23/2021, 4:49 PM
@nicholas 0.14.12
n

nicholas

03/23/2021, 4:50 PM
Great, thank you - we're looking into this, I'll let you know as soon as we have an answer!
🙏 2
m

Mariia Kerimova

03/23/2021, 11:23 PM
Hello Renzo! i was able to replicate the error you are having. The trick is to adjust the permission policy you provide to this user. Some actions such as
ecs:RegisterTaskDefinition
can’t be set only on specific cluster. You might have to set conditions like here:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ecs:PutAttributes",
        "ecs:ListAttributes",
        "ecs:UpdateContainerInstancesState",
        "ecs:StartTask",
        "ecs:RegisterContainerInstance",
        "ecs:DeleteAttributes",
        "ecs:DescribeTaskSets",
        "ecs:DeleteCapacityProvider",
        "ecs:SubmitAttachmentStateChanges",
        "ecs:Poll",
        "ecs:UpdateService",
        "ecs:DescribeCapacityProviders",
        "ecs:CreateService",
        "ecs:RunTask",
        "ecs:ListTasks",
        "ecs:StopTask",
        "ecs:DescribeServices",
        "ecs:SubmitContainerStateChange",
        "ecs:DescribeContainerInstances",
        "ecs:DeregisterContainerInstance",
        "ecs:TagResource",
        "ecs:DescribeTasks",
        "ecs:UntagResource",
        "ecs:PutClusterCapacityProviders",
        "ecs:UpdateTaskSet",
        "ecs:SubmitTaskStateChange",
        "ecs:UpdateClusterSettings",
        "ecs:DeleteService",
        "ecs:DeleteCluster",
        "ecs:DeleteTaskSet",
        "ecs:DescribeClusters",
        "ecs:ListTagsForResource",
        "ecs:StartTelemetrySession",
        "ecs:UpdateContainerAgent",
        "ecs:ListContainerInstances",
        "ecs:UpdateServicePrimaryTaskSet"
      ],
      "Resource": [
        "arn:aws:ecs:us-east-1:<account>:cluster/<cluster_name>",
        "arn:aws:ecs:us-east-1:<account>:task-definition/prefect-<flow_name>:*"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "ecs:DiscoverPollEndpoint",
        "ecs:PutAccountSettingDefault",
        "ecs:CreateCluster",
        "ecs:DescribeTaskDefinition",
        "ecs:PutAccountSetting",
        "ecs:ListServices",
        "ecs:CreateCapacityProvider",
        "ecs:DeregisterTaskDefinition",
        "ecs:ListAccountSettings",
        "ecs:DeleteAccountSetting",
        "ecs:ListTaskDefinitionFamilies",
        "ecs:RegisterTaskDefinition",
        "ecs:ListTaskDefinitions",
        "ecs:CreateTaskSet",
        "ecs:ListClusters"
      ],
      "Condition": {
        "StringLike": {
          "aws:TagKeys": [
            "prefect:flow-id"
          ]
        }
      }
    }
  ]
}
In short, you need to update the policy, and it should solve the issue. Also, you can find additional information in AWS documentation
r

Renzo Becerra

03/23/2021, 11:29 PM
@Mariia Kerimova Thank you for the sharing this! Will I need to include additional policies other than what you have sent me above?
m

Mariia Kerimova

03/23/2021, 11:39 PM
This is a list of main permissions you need to provide, but you might need to add more permissions to the policy, but it’s hard to say without looking at your setup
Feel free to ping me if you’ll run into errors, after you update the policy 🙂
r

Renzo Becerra

03/25/2021, 3:47 PM
@Mariia Kerimova Can you please send us the necessary policies for the ecs cluster? and the policies required for an ec2 agent
m

Mariia Kerimova

03/26/2021, 6:20 PM
Hello Renzo. Sorry for long reply. This is minimal policy, but also checkout policies for task-role and execution role in our docs.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "ec2:DeleteSecurityGroup",
                "ecs:CreateCluster",
                "ecs:DeleteCluster",
                "ecs:DeregisterTaskDefinition",
                "ecs:DescribeClusters",
                "ecs:DescribeTaskDefinition",
                "ecs:DescribeTasks",
                "ecs:ListAccountSettings",
                "ecs:ListClusters",
                "ecs:ListTaskDefinitions",
                "ecs:RegisterTaskDefinition",
                "ecs:RunTask",
                "ecs:StopTask",
                "iam:AttachRolePolicy",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:DetachRolePolicy",
                "iam:ListAttachedRolePolicies",
                "iam:ListInstanceProfiles",
                "iam:ListRolePolicies",
                "iam:ListRoles",
                "iam:ListRoleTags",
                "iam:TagRole",
                "logs:DescribeLogGroups",
                "logs:GetLogEvents"
            ],
            "Resource": "*"
        }
}