https://prefect.io logo
Channels
Powered by
a

ale

10/05/2020, 1:49 PM
Hey folks, finally I was able to configure Fargate Agent. Now I want to run a task on Fargate. I’m using the following config:
Copy code
FargateTaskEnvironment(
    taskRoleArn=ETL_TASK_ROLE_ARN,  # ARN of the task role
    executionRoleArn=ETL_EXECUTION_ROLE_ARN  #  ARN of the execution role
)
But I get the following error back from the Agent:
Copy code
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images
Any suggestions?
s

Spencer

10/05/2020, 2:36 PM
I'd be curious about the roles and if they have proper policies
a

ale

10/05/2020, 2:51 PM
The execution role has this policy attached
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
The Fargate Agent (which runs itself on Fargate) has the following policy:
Copy code
- Sid: ResourceAllItems
            Effect: Allow
            Action:
              - tag:Get*
              - logs:PutLogEvents
              - logs:CreateLogStream
              - logs:CreateLogGroup
              - events:PutTargets
              - events:PutRule
              - events:DescribeRule
              - ecs:StopTask
              - ecs:RegisterTaskDefinition
              - ecs:Describe*
              - ecr:GetDownloadUrlForLayer
              - ecr:GetAuthorizationToken
              - ecr:BatchGetImage
              - ecr:BatchCheckLayerAvailability
              - ec2:DescribeSubnets
            Resource: '*'
s

Spencer

10/05/2020, 3:01 PM
I see
How is the agent configured? What 
executionRoleArn
has been given to the agent's configuration? (This is different from the 
executionRoleArn
used to run the agent, if running it on Fargate)\
a

ale

10/05/2020, 3:14 PM
I think I did not understand your question 😓 Can you re-explain please?
Sorry I made a mistake. Fargate Agent runs on ECS, but with EC2 launch type.
s

Spencer

10/05/2020, 3:21 PM
How is the agent configured?
a

ale

10/05/2020, 3:25 PM
This is the agent configuration
Copy code
# Prefect Fargate Agent
  rAgentTaskRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Statement:
        - Effect: 'Allow'
          Principal:
            Service: ['<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>']
          Action: ['sts:AssumeRole']
      Path: /

  rAgentTaskRolePolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: ResourceAllItems
            Effect: Allow
            Action:
              - tag:Get*
              - logs:PutLogEvents
              - logs:CreateLogStream
              - logs:CreateLogGroup
              - events:PutTargets
              - events:PutRule
              - events:DescribeRule
              - ecs:StopTask
              - ecs:RegisterTaskDefinition
              - ecs:Describe*
              - ecr:GetDownloadUrlForLayer
              - ecr:GetAuthorizationToken
              - ecr:BatchGetImage
              - ecr:BatchCheckLayerAvailability
              - ec2:DescribeSubnets
            Resource: '*'
          - Sid: EcsTaskRun
            Effect: Allow
            Action: ecs:RunTask
            Resource: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task-definition/*
            Condition:
              ForAllValues:StringEquals:
                aws:TagKeys:
                  - PrefectFlowVersion
                  - PrefectFlowId
          - Sid: IamPassRole
            Effect: Allow
            Action: iam:PassRole
            Resource: '*'
      Roles:
        - !Ref rAgentTaskRole

  rAgentTaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E1029
    Properties:
      TaskRoleArn: !Ref 'rAgentTaskRole'
      Family: !Sub '${pEnvironment}-prefect-agent'
      ContainerDefinitions:
        - Name: 'agent'
          Essential: true
          Command: ['prefect', 'agent', 'start', 'fargate', '--verbose', 'enable_task_revisions=true']
          Image: !Sub
            - 'prefecthq/prefect:${version}'
            - version: !If [ cIsPrefectServerVersionLatest, 'all_extras', !Sub 'all_extras-${pPrefectServerVersion}' ]
          MemoryReservation: 128
          Environment:
            - Name: PREFECT__CLOUD__API
              Value: '<obfuscated>'
            - Name: PREFECT__BACKEND
              Value: 'server'
            - Name: REGION_NAME
              Value: !Ref AWS::Region
            - Name: networkConfiguration
              Value: !Sub
                - '{"awsvpcConfiguration": {"assignPublicIp": "ENABLED", "subnets": ["${subnets}"], "securityGroups": ["${securityGroups}"]}}'
                - subnets: !Join ['","', !Ref pFargateSubnetIds]
                  securityGroups: !Join ['","', !Ref pFargateSecurityGroupIds]
            - Name: cpu
              Value: 256
            - Name: memory
              Value: 1024
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              awslogs-group: !Ref 'rAppLogsGroup'
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: 'prefect-agent'
      Tags:
        - Key: 'Name'
          Value: !Sub '${pEnvironment}-prefect-agent'
        - Key: 'ApplicationVersion'
          Value: !Ref 'pApplicationVersion'

  rAgentEcsService:
    Type: 'AWS::ECS::Service'
    Properties:
      Cluster: !Ref 'pEcsCluster'
      DesiredCount: 1
      TaskDefinition: !Ref 'rAgentTaskDefinition'
      PropagateTags: 'TASK_DEFINITION'
      EnableECSManagedTags: true
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
      PlacementStrategies:
      - Type: 'spread'
        Field: 'attribute:ecs.availability-zone'
      Tags:
      - Key: 'Name'
        Value: !Sub '${pEnvironment}-prefect-agent'
      - Key: 'ApplicationVersion'
        Value: !Ref 'pApplicationVersion'
s

Spencer

10/05/2020, 3:29 PM
See, the agent has no executionRoleArn in it's ENV
The Fargate agent first creates a task that will, in turn, download the configuration for the flow from the prefect API and run the flow in another task on Fargate. This first task is created by the agent and uses the executionRoleArn and taskRoleArn from the Agent's configuration.
a

ale

10/05/2020, 3:46 PM
@Giacomo Consonni
Thanks for the explanation @Spencer
Btw, is this behaviour explained in the docs? I think I’ve not found anything similar to your explanation in the official docs
s

Spencer

10/05/2020, 5:13 PM
I don't think so; I figured it out from running it myself 🤷‍♂️
a

ale

10/05/2020, 6:46 PM
Good to know! Maybe someone from Prefect can take care of updating the docs with your findings :)
g

Giacomo Consonni

10/05/2020, 8:21 PM
@Spencer I tested what you proposed but it seems nothing changed. Now the agent has this execution role:
Copy code
rExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
      AssumeRolePolicyDocument:
        Statement:
          - Effect: 'Allow'
            Principal:
              Service: ['<http://ecs-tasks.amazonaws.com|ecs-tasks.amazonaws.com>']
            Action: ['sts:AssumeRole']
      Path: /
The 
AmazonECSTaskExecutionRolePolicy
grant these permissions:
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}
Starting a flow results in this error:
Copy code
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Fargate requires task definition to have execution role ARN to support ECR images.
Looking at Cloudtrail the only thing I can see is:
Copy code
{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAWENMXISUN465BMEDP:b52b5f006e154dc7afdda3f63c7c2861",
    "arn": "arn:aws:sts::421805900968:assumed-role/stage-safelake-etl-rAgentTaskRole-1JTX5S73405MQ/b52b5f006e154dc7afdda3f63c7c2861",
    "accountId": "421805900968",
    "accessKeyId": "ASIAWENMXISUJJFUMP5K",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROAWENMXISUN465BMEDP",
        "arn": "arn:aws:iam::421805900968:role/stage-safelake-etl-rAgentTaskRole-1JTX5S73405MQ",
        "accountId": "421805900968",
        "userName": "stage-safelake-etl-rAgentTaskRole-1JTX5S73405MQ"
      },
      "webIdFederationData": {},
      "attributes": {
        "mfaAuthenticated": "false",
        "creationDate": "2020-10-05T16:05:00Z"
      }
    }
  },
  "eventTime": "2020-10-05T20:12:16Z",
  "eventSource": "<http://ecs.amazonaws.com|ecs.amazonaws.com>",
  "eventName": "RegisterTaskDefinition",
  "awsRegion": "us-west-2",
  "sourceIPAddress": "52.39.51.8",
  "userAgent": "Boto3/1.15.7 Python/3.8.6 Linux/4.14.193-149.317.amzn2.x86_64 exec-env/AWS_ECS_EC2 Botocore/1.18.7",
  "errorCode": "ClientException",
  "errorMessage": "Fargate requires task definition to have execution role ARN to support ECR images.",
  "requestParameters": null,
  "responseElements": null,
  "requestID": "29e54079-7894-4709-86c7-20961d1135e4",
  "eventID": "42558309-7e3b-4990-8f8e-116e5e73b3ee",
  "eventType": "AwsApiCall",
  "recipientAccountId": "421805900968"
}
But I can't find any call to the RunTask API. It seems to me that during the registration phase of the task neither e execution role arn and task role arn are correctly registered in the flow defintion. This is the registration script
Copy code
import os
import prefect
from prefect import Flow
from prefect.environments.storage import Docker
from prefect.environments.execution import FargateTaskEnvironment
from slugify import slugify

ETL_FLOW_NAME = os.environ["ETL_FLOW_NAME"]
ETL_IMAGE_NAME = os.environ["ETL_IMAGE_NAME"]
ETL_IMAGE_TAG = os.environ["ETL_IMAGE_TAG"]
ETL_PROJECT_NAME = os.environ["ETL_PROJECT_NAME"]
ETL_TASK_ROLE_ARN = os.environ["ETL_TASK_ROLE_ARN"]
ETL_EXECUTION_ROLE_ARN = os.environ["ETL_EXECUTION_ROLE_ARN"]


storage = Docker(
    image_name=ETL_IMAGE_NAME,
    image_tag=ETL_IMAGE_TAG
)
environment = FargateTaskEnvironment(
    cpu="256",
    memory="512",
    taskRoleArn=ETL_TASK_ROLE_ARN,
    executionRoleArn=ETL_EXECUTION_ROLE_ARN,
)


def main():
    flow = Flow.load(fpath=f"{prefect.context.config.home_dir}/flows/{slugify(ETL_FLOW_NAME)}.prefect")
    flow.environment = environment
    flow.storage = storage
    storage.add_flow(flow=flow)
    flow.register(
        project_name=ETL_PROJECT_NAME,
        build=False
    )


if __name__ == '__main__':
    main()
s

Spencer

10/05/2020, 8:22 PM
You misunderstood me: There are three locations where the executionRoleArn is placed when running the Fargate Agent on Fargate itself. 1. The task definition of the Prefect Agent 2. The configuration of the Fargate Agent (via ENV or CLI args) 3. The FargateTaskEnvironment
g

Giacomo Consonni

10/05/2020, 8:22 PM
Environment variable are correct and the registrarion process returns a positive result
Ok, I think the second one is missing. What do you mean exactly? Which role should I pass as an environment variable to the Fargate agent? Which name should I use?
(by the way, thank you for your so quick reply!)
s

Spencer

10/05/2020, 8:27 PM
You've configured the networkConfiguration and cpu/memory properly, these ARNs go there in the ENV of the Agent itself
g

Giacomo Consonni

10/05/2020, 8:30 PM
Ok, so now the task definition of the agents looks like this:
Copy code
rAgentTaskDefinition:
    Type: 'AWS::ECS::TaskDefinition'
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E1029
    Properties:
      TaskRoleArn: !Ref 'rAgentTaskRole'
      ExecutionRoleArn: !GetAtt 'rExecutionRole.Arn'
      Family: !Sub '${pEnvironment}-prefect-agent'
      ContainerDefinitions:
        - Name: 'agent'
          Essential: true
          Command: ['prefect', 'agent', 'start', 'fargate', '--verbose', 'enable_task_revisions=true']
          Image: !Sub
            - 'prefecthq/prefect:${version}'
            - version: !If [ cIsPrefectServerVersionLatest, 'all_extras', !Sub 'all_extras-${pPrefectServerVersion}' ]
          MemoryReservation: 128
          Environment:
            - Name: PREFECT__CLOUD__API
              Value: '<https://apollo-prefect-server.srv-stage.cloudacademy.xyz/graphql>'
            - Name: PREFECT__BACKEND
              Value: 'server'
            - Name: REGION_NAME
              Value: !Ref AWS::Region
            - Name: executionRoleArn
              Value: !GetAtt 'rExecutionRole.Arn'
            - Name: networkConfiguration
              Value: !Sub
                - '{"awsvpcConfiguration": {"assignPublicIp": "ENABLED", "subnets": ["${subnets}"], "securityGroups": ["${securityGroups}"]}}'
                - subnets: !Join ['","', !Ref pFargateSubnetIds]
                  securityGroups: !Join ['","', !Ref pFargateSecurityGroupIds]
            - Name: cpu
              Value: 256
            - Name: memory
              Value: 512
          LogConfiguration:
            LogDriver: 'awslogs'
            Options:
              awslogs-group: !Ref 'rAppLogsGroup'
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: 'prefect-agent'
I added the 
executionRoleArn
environment variable with value the same of the execution role arn assumed by the agent task. Should I add also the taskRoleArn?
s

Spencer

10/05/2020, 8:40 PM
I believe so. I haven't investigated the necessity of it, but I did it. 🤷‍♂️
g

Giacomo Consonni

10/05/2020, 8:41 PM
Ok now it worked. In this way how can I set different execution and task roles for each flow? It seems that the only role that a Fargate task can assume is the one I passed as an environment variable in the agent configuration. It seems the class 
FargateTaskEnvironment
handle the 2 parameters: 
taskRoleArn
and 
executionRoleArn
but are they useless?
s

Spencer

10/05/2020, 8:41 PM
No
You've misunderstood how the Fargate agent works
The Fargate agent itself doesn't have to run on Fargate and that's why it's confusing.
g

Giacomo Consonni

10/05/2020, 8:43 PM
The agent is not running on fargate but is running as EC2 task in ECS
s

Spencer

10/05/2020, 8:43 PM
sure, that's not important
The Fargate agent polls the API and kicks off a bootstrap task (task #2) on Fargate to get the configuration which will in turn run the flow in another task (task #3)
The ARNs that you added are for Task #2 which only queries the API and triggers 
ecs:RegisterTaskDefinition
and 
ecs:RunTask
The roles of your flow and prefect.Tasks are the roles assigned to the ECS Task #3, these are configured in the FargateTaskEnvironment defined in your flows
If you want different roles for each flow, you must associate different `FargateTaskEnvironment`s for each flow with the roles that you desire
Am I making sense?
g

Giacomo Consonni

10/05/2020, 8:50 PM
Oh my god! Now it really makes sense. Thank you 😄
Powered by