https://prefect.io logo
Title
a

Abhishek Mitra

02/01/2023, 4:09 AM
Hello everyone, I have been trying to run a flow through an ECS Agent deployed in a fargate cluster. I am following this document for setting it up. But I keep getting the below error. I checked the all the roles and attached policies and all seems good. Does anybody know how to debug this?
Watching flow run execution...
└── 16:18:49 | INFO    | Entered state <Scheduled>: Flow run scheduled.
└── 16:18:57 | INFO    | Entered state <Submitted>: Submitted for execution
└── 16:18:57 | ERROR   | An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid
└── 16:18:58 | INFO    | Entered state <Failed>: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid
Flow run failed!
Can anybody help me out here please? I haven't been able to make any progress on this for a long time.
1
c

Claire Herdeman

02/01/2023, 3:39 PM
We also use ECS Agent on a Fargate cluster, it looks like the issue is with your IAM task or execution role for your agent. I haven't seen "Role is not valid", have you taken a look in at your task and execution roles in the UI to see if your role definitions and permissions are correctly formatted? Do you have the Register Task Definition permission associated with the correct role?
:thank-you: 1
a

Abhishek Mitra

02/01/2023, 3:45 PM
These are the policies attached to the role I'm using : prefectTaskRolePolicy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateSecurityGroup",
        "ec2:CreateTags",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:DeleteSecurityGroup",
        "ecs:CreateCluster",
        "ecs:DeleteCluster",
        "ecs:DeregisterTaskDefinition",
        "ecs:DescribeClusters",
        "ecs:DescribeTaskDefinition",
        "ecs:DescribeTasks",
        "ecs:ListAccountSettings",
        "ecs:ListClusters",
        "ecs:ListTaskDefinitions",
        "ecs:RegisterTaskDefinition",
        "ecs:RunTask",
        "ecs:StopTask",
        "iam:PassRole",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
prefectTaskRoleS3Policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": "arn:aws:s3:::*prefect*"
    }
  ]
}
Also won't aws throw a error if the attached policy is wrong. I think they review it before saving
Jfyi, I also used policysim to test if RegisterTaskDefinition was accessible.. and it was. So now I don't even know where to start looking.
c

Carlo

02/01/2023, 4:08 PM
The task definition has two roles, an executionRole and taskRole. I would verify what you are actually submitting to aws, and ensure each has the appropriate permission.
When I need more insight, I turn off all ecs agents, and just run it locally. So I can debug the values being submitted to aws via boto.
like so
💡 1
a

Abhishek Mitra

02/01/2023, 4:54 PM
@Carlo The above code uses prefect 2.x but I'm using 1.2.2. Can you suggest something for prefect v1? Also I checked execution role arn and task role arn for agent, prefect flow and aws, all three are good, no typo.
c

Claire Herdeman

02/01/2023, 5:05 PM
Hmm, https://github.com/aws-samples/aws-modern-application-workshop/issues/9, do you have a trust relationship on the task role? I'll same i'm also using prefect 2 but mine looks like this:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
c

Carlo

02/01/2023, 6:18 PM
@Abhishek Mitra you have an agent running somewhere w/ your v1 code. Figure out which module is used, and just call that locally
c

Christopher Boyd

02/02/2023, 2:09 PM
Echoing @Carlo above, this is an AWS issue with your roles. I would - 1. Make sure the role is in the same account where you are trying to execute runTask 2. Make sure the role has ecs-tasks trust relationship 3. Make sure the role has required permissions 4. Check your execution role, as well as your task role as they are separate. https://discourse.prefect.io/t/ecs-task-role-set-on-ecs-agent-startup-command-isnt-applied-to-flows-run-configuration/766/3