Hey folks, finally I was able to configure Fargate Agent. Now I want to run a task on Fargate. I’m using the following config:

FargateTaskEnvironment(
    taskRoleArn=ETL_TASK_ROLE_ARN,  # ARN of the task role
    executionRoleArn=ETL_EXECUTION_ROLE_ARN  #  ARN of the execution role
)

But I get the following error back from the Agent:

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images

Any suggestions?

I'd be curious about the roles and if they have proper policies

The execution role has this policy attached

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

I see

I think I did not understand your question 😓 Can you re-explain please?

How is the agent configured?

This is the agent configuration

# Prefect Fargate Agent
  rAgentTaskRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: 'Allow'
          Principal:
            Service: ['<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>']
          Action: ['sts:AssumeRole']
      Path: /

  rAgentTaskRolePolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: ResourceAllItems
            Effect: Allow
            Action:
              - tag:Get*
              - logs:PutLogEvents
              - logs:CreateLogStream
              - logs:CreateLogGroup
              - events:PutTargets
              - events:PutRule
              - events:DescribeRule
              - ecs:StopTask
              - ecs:RegisterTaskDefinition
              - ecs:Describe*
              - ecr:GetDownloadUrlForLayer
              - ecr:GetAuthorizationToken
              - ecr:BatchGetImage
              - ecr:BatchCheckLayerAvailability
              - ec2:DescribeSubnets
            Resource: '*'
          - Sid: EcsTaskRun
            Effect: Allow
            Action: ecs:RunTask
            Resource: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task-definition/*
            Condition:
              ForAllValues:StringEquals:
                aws:TagKeys:
                  - PrefectFlowVersion
                  - PrefectFlowId
          - Sid: IamPassRole
            Effect: Allow
            Action: iam:PassRole
            Resource: '*'
      Roles:
        - !Ref rAgentTaskRole

  rAgentTaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E1029
    Properties:
      TaskRoleArn: !Ref 'rAgentTaskRole'
      Family: !Sub '${pEnvironment}-prefect-agent'
      ContainerDefinitions:
        - Name: 'agent'
          Essential: true
          Command: ['prefect', 'agent', 'start', 'fargate', '--verbose', 'enable_task_revisions=true']
          Image: !Sub
            - 'prefecthq/prefect:${version}'
            - version: !If [ cIsPrefectServerVersionLatest, 'all_extras', !Sub 'all_extras-${pPrefectServerVersion}' ]
          MemoryReservation: 128
          Environment:
            - Name: PREFECT__CLOUD__API
              Value: '<obfuscated>'
            - Name: PREFECT__BACKEND
              Value: 'server'
            - Name: REGION_NAME
              Value: !Ref AWS::Region
            - Name: networkConfiguration
              Value: !Sub
                - '{"awsvpcConfiguration": {"assignPublicIp": "ENABLED", "subnets": ["${subnets}"], "securityGroups": ["${securityGroups}"]}}'
                - subnets: !Join ['","', !Ref pFargateSubnetIds]
                  securityGroups: !Join ['","', !Ref pFargateSecurityGroupIds]
            - Name: cpu
              Value: 256
            - Name: memory
              Value: 1024
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              awslogs-group: !Ref 'rAppLogsGroup'
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: 'prefect-agent'
      Tags:
        - Key: 'Name'
          Value: !Sub '${pEnvironment}-prefect-agent'
        - Key: 'ApplicationVersion'
          Value: !Ref 'pApplicationVersion'

  rAgentEcsService:
    Type: 'AWS::ECS::Service'
    Properties:
      Cluster: !Ref 'pEcsCluster'
      DesiredCount: 1
      TaskDefinition: !Ref 'rAgentTaskDefinition'
      PropagateTags: 'TASK_DEFINITION'
      EnableECSManagedTags: true
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      PlacementStrategies:
      - Type: 'spread'
        Field: 'attribute:ecs.availability-zone'
      Tags:
      - Key: 'Name'
        Value: !Sub '${pEnvironment}-prefect-agent'
      - Key: 'ApplicationVersion'
        Value: !Ref 'pApplicationVersion'

See, the agent has no executionRoleArn in it's ENV

I don't think so; I figured it out from running it myself 🤷‍♂️

Good to know! Maybe someone from Prefect can take care of updating the docs with your findings :)

@Spencer I tested what you proposed but it seems nothing changed. Now the agent has this execution role:

rExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
      AssumeRolePolicyDocument:
        Statement:
          - Effect: 'Allow'
            Principal:
              Service: ['<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>']
            Action: ['sts:AssumeRole']
      Path: /

The

AmazonECSTaskExecutionRolePolicy

grant these permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Starting a flow results in this error:

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.

Looking at Cloudtrail the only thing I can see is:

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAWENMXISUN465BMEDP:b52b5f006e154dc7afdda3f63c7c2861",
    "arn": "arn:aws:sts::421805900968:assumed-role/stage-safelake-etl-rAgentTaskRole-1JTX5S73405MQ/b52b5f006e154dc7afdda3f63c7c2861",
    "accountId": "421805900968",
    "accessKeyId": "ASIAWENMXISUJJFUMP5K",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROAWENMXISUN465BMEDP",
        "arn": "arn:aws:iam::421805900968:role/stage-safelake-etl-rAgentTaskRole-1JTX5S73405MQ",
        "accountId": "421805900968",
        "userName": "stage-safelake-etl-rAgentTaskRole-1JTX5S73405MQ"
      },
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "false",
        "creationDate": "2020-10-05T16:05:00Z"
      }
    }
  },
  "eventTime": "2020-10-05T20:12:16Z",
  "eventSource": "<http://ecs.amazonaws.com|ecs.amazonaws.com>",
  "eventName": "RegisterTaskDefinition",
  "awsRegion": "us-west-2",
  "sourceIPAddress": "52.39.51.8",
  "userAgent": "Boto3/1.15.7 Python/3.8.6 Linux/4.14.193-149.317.amzn2.x86_64 exec-env/AWS_ECS_EC2 Botocore/1.18.7",
  "errorCode": "ClientException",
  "errorMessage": "Fargate requires task definition to have execution role ARN to support ECR images.",
  "requestParameters": null,
  "responseElements": null,
  "requestID": "29e54079-7894-4709-86c7-20961d1135e4",
  "eventID": "42558309-7e3b-4990-8f8e-116e5e73b3ee",
  "eventType": "AwsApiCall",
  "recipientAccountId": "421805900968"
}

But I can't find any call to the RunTask API. It seems to me that during the registration phase of the task neither e execution role arn and task role arn are correctly registered in the flow defintion. This is the registration script

You misunderstood me: There are three locations where the executionRoleArn is placed when running the Fargate Agent on Fargate itself. 1. The task definition of the Prefect Agent 2. The configuration of the Fargate Agent (via ENV or CLI args) 3. The FargateTaskEnvironment

Environment variable are correct and the registrarion process returns a positive result

You've configured the networkConfiguration and cpu/memory properly, these ARNs go there in the ENV of the Agent itself

Ok, so now the task definition of the agents looks like this:

rAgentTaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E1029
    Properties:
      TaskRoleArn: !Ref 'rAgentTaskRole'
      ExecutionRoleArn: !GetAtt 'rExecutionRole.Arn'
      Family: !Sub '${pEnvironment}-prefect-agent'
      ContainerDefinitions:
        - Name: 'agent'
          Essential: true
          Command: ['prefect', 'agent', 'start', 'fargate', '--verbose', 'enable_task_revisions=true']
          Image: !Sub
            - 'prefecthq/prefect:${version}'
            - version: !If [ cIsPrefectServerVersionLatest, 'all_extras', !Sub 'all_extras-${pPrefectServerVersion}' ]
          MemoryReservation: 128
          Environment:
            - Name: PREFECT__CLOUD__API
              Value: '<https://apollo-prefect-server.srv-stage.cloudacademy.xyz/graphql>'
            - Name: PREFECT__BACKEND
              Value: 'server'
            - Name: REGION_NAME
              Value: !Ref AWS::Region
            - Name: executionRoleArn
              Value: !GetAtt 'rExecutionRole.Arn'
            - Name: networkConfiguration
              Value: !Sub
                - '{"awsvpcConfiguration": {"assignPublicIp": "ENABLED", "subnets": ["${subnets}"], "securityGroups": ["${securityGroups}"]}}'
                - subnets: !Join ['","', !Ref pFargateSubnetIds]
                  securityGroups: !Join ['","', !Ref pFargateSecurityGroupIds]
            - Name: cpu
              Value: 256
            - Name: memory
              Value: 512
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              awslogs-group: !Ref 'rAppLogsGroup'
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: 'prefect-agent'

I added the

executionRoleArn

environment variable with value the same of the execution role arn assumed by the agent task. Should I add also the taskRoleArn?

I believe so. I haven't investigated the necessity of it, but I did it. 🤷‍♂️

Ok now it worked. In this way how can I set different execution and task roles for each flow? It seems that the only role that a Fargate task can assume is the one I passed as an environment variable in the agent configuration. It seems the class

FargateTaskEnvironment

handle the 2 parameters:

taskRoleArn

and

executionRoleArn

but are they useless?

No

The agent is not running on fargate but is running as EC2 task in ECS

sure, that's not important

Oh my god! Now it really makes sense. Thank you 😄