https://prefect.io logo
Channels
Powered by
#prefect-server
Title
# prefect-server
k

Karl

05/19/2021, 12:41 AM
Hello all, I’m attempting to stand up a POC in the AWS cloud using AWS ECS Fargate and Prefect Cloud and hit a roadblock. Error: "when calling the RegisterTaskDefinition operation: Role is not valid" It appears to be associated with my AWS roles/permission but I’m not sure. Not sure what I am doing wrong..
n

nicholas

05/19/2021, 12:52 AM
Hi @Karl - can you move all but your initial question into this thread? Long messages tend to crowd out other messages and make it more difficult for everyone to read messages in the channel.
k

Karl

05/19/2021, 2:08 AM
Here are my current configuration and environment set up for Prefect, AWS VPC, Fargate, script and steps. *Sensitive information has been masked by an XXXXXXXXXXX. Prefect diagnostics:
Copy code
{
  "config_overrides": {
    "cloud": {
      "agent": {
        "auth_token": true
      },
      "auth_token": true
    }
  },
  "env_vars": [],
  "system_information": {
    "platform": "Windows-10-10.0.18362-SP0",
    "prefect_backend": "cloud",
    "prefect_version": "0.14.19",
    "python_version": "3.8.8"
  }
}
I used the following guide to configure the docker storage and AWS roles. Many thanks to Jimmy Le! https://www.lejimmy.com/distributed-data-pipelines-with-aws-ecs-fargate-and-prefect-cloud/ Even created a ecs-admin role as described in the former guide. With one exception, I created a AWS ECS Fargate cluster in the AWS managment console. Which configured a default VPC ( the only VPC on my AWS account ). First, I started my ECS Agent from my local machine Spyder IDE.
Copy code
from prefect.agent.ecs.agent import ECSAgent

AGENT = ECSAgent(cluster='arn:aws:ecs:us-west-2:XXXXXXXXXXX:cluster/test-cluster',
                  #task_role_arn='arn:aws:iam::XXXXXXXXXXX:role/ecs-admin',
                  labels=['dev'])
AGENT.start()
The agent is recognized by the Prefect Cloud UI. Then, from my local machine IDE I call script flow2.py. -flow2py:
Copy code
import prefect
from prefect.storage import Docker
from prefect.run_configs import ECSRun
from prefect import task, Flow
#import yaml

STORAGE = Docker(registry_url='<http://XXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/|XXXXXXXXXXX.dkr.ecr.us-west-2.amazonaws.com/>', 
                 image_name='oceancluster-docker', 
                 image_tag='latest',
                 dockerfile=r"C:\Users\XXXXXXXXXXX\cluster01-docker\Dockerfile.txt")
    
# definition = yaml.safe_load(
#     """
#     networkMode: awsvpc
#     cpu: 512
#     memory: 256
#     containerDefinitions:
#         - name: flow
#     requiresCompatibilities:
#         - FARGATE
#     execution_role_arn: arn:aws:iam::XXXXXXXXXXX:role/ecsTaskExecutionRole
#     """
# )
    
    
RUN_CONFIG = ECSRun(
    run_task_kwargs = {
                 "taskDefinition":"14-fargate",
                 'cluster':'arn:aws:ecs:us-west-2:XXXXXXXXXXX:cluster/test-cluster',
                 'region_name':'us-west-2',
                 'networkConfiguration': {'awsvpcConfiguration': {'assignPublicIp': 'ENABLED', 'subnets': ['subnet-06eadc5f89d2293b2','subnet-0a2e199d91e06a92b'], 'securityGroups': []}},
                 'launchType':'FARGATE'
                 },
                    execution_role_arn='arn:aws:iam::XXXXXXXXXXX:role/ecs-admin',
                    #image='prefecthq/prefect:latest-python3.8',
                    memory='512',
                    cpu='256',
                    labels=['dev'])

#EXECUTOR = DaskExecutor()

@task
def say_hello():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Karl! Good Job, it worked@@!")

with Flow('hello-flow', storage=STORAGE, run_config=RUN_CONFIG ) as flow:
    say_hello()
    
# Register the flow under the "tutorial" project
flow.register(project_name="test")
Which builds and pushes my docker image to AWS ECR. I can manually see the image when I log into my AWS ECR repository. Logging into the Prefect cloud, I see my flow and task. However, when I RUN my flow, I am presented with the following error message in my console and Prefect UI.
Copy code
[2021-05-18 23:36:53,201] INFO - agent | Starting ECSAgent with labels ['dev']
[2021-05-18 23:36:53,202] INFO - agent | Agent documentation can be found at <https://docs.prefect.io/orchestration/>
[2021-05-18 23:36:53,204] INFO - agent | Agent connecting to the Prefect API at <https://api.prefect.io>
[2021-05-18 23:36:53,386] INFO - agent | Waiting for flow runs...
[2021-05-18 23:40:54,356] INFO - agent | Found 1 flow run(s) to submit for execution.
[2021-05-18 23:40:54,587] INFO - agent | Deploying flow run '5d5df04b-96b0-4128-bb73-e3c2b8a2ab0d'
[2021-05-18 23:40:54,833] ERROR - agent | Logging platform error for flow run 5d5df04b-96b0-4128-bb73-e3c2b8a2ab0d
[2021-05-18 23:40:55,163] ERROR - agent | Error while deploying flow
Traceback (most recent call last):
  File "C:\Users\XXXXXXXXXXX\Anaconda3\envs\POC\lib\site-packages\prefect\agent\agent.py", line 420, in deploy_and_update_flow_run
    deployment_info = self.deploy_flow(flow_run)
  File "C:\Users\XXXXXXXXXXX\Anaconda3\envs\POC\lib\site-packages\prefect\agent\ecs\agent.py", line 297, in deploy_flow
    resp = self.ecs_client.register_task_definition(**taskdef)
  File "C:\Users\XXXXXXXXXXX\Anaconda3\envs\POC\lib\site-packages\botocore\client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\XXXXXXXXXXX\Anaconda3\envs\POC\lib\site-packages\botocore\client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid
Attached are screenshots of my AWS role*, VPC, and ECS Fargate configurations: *appears that AWS no longer allows "AWSServiceRoleForECS" to be added to a role as illustrated in the guide.
Copy code
{
  "Stacks": [
    {
      "StackId": "arn:aws:cloudformation:us-west-2:XXXXXXXXXXX:stack/EC2ContainerService-test-cluster/69ab9fc0-b803-11eb-bcf3-06c940f8e023",
      "StackName": "EC2ContainerService-test-cluster",
      "Description": "AWS CloudFormation template to create a new VPC to work with Fargate in Create Cluster Wizard\n",
      "Parameters": [
        {
          "ParameterKey": "SecurityIngressToPort",
          "ParameterValue": "65535"
        },
        {
          "ParameterKey": "EcsEndpoint",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "IsFargate",
          "ParameterValue": "true"
        },
        {
          "ParameterKey": "IamRoleInstanceProfile",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "EbsVolumeSize",
          "ParameterValue": "0"
        },
        {
          "ParameterKey": "SecurityIngressFromPort",
          "ParameterValue": "0"
        },
        {
          "ParameterKey": "SecurityGroupId",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "EcsClusterName",
          "ParameterValue": "test-cluster"
        },
        {
          "ParameterKey": "SubnetIds",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "KeyName",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "SubnetCidr3",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "SubnetCidr2",
          "ParameterValue": "10.0.1.0/24"
        },
        {
          "ParameterKey": "SubnetCidr1",
          "ParameterValue": "10.0.0.0/24"
        },
        {
          "ParameterKey": "SecurityIngressCidrIp",
          "ParameterValue": "0.0.0.0/0"
        },
        {
          "ParameterKey": "VpcId",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "VpcCidr",
          "ParameterValue": "10.0.0.0/16"
        },
        {
          "ParameterKey": "AsgMaxSize",
          "ParameterValue": "1"
        },
        {
          "ParameterKey": "EcsInstanceType",
          "ParameterValue": "m4.large"
        },
        {
          "ParameterKey": "EbsVolumeType",
          "ParameterValue": ""
        },
        {
          "ParameterKey": "EcsAmiId",
          "ParameterValue": "ami-084799b9fb64c149e"
        },
        {
          "ParameterKey": "VpcAvailabilityZones",
          "ParameterValue": "us-west-2a,us-west-2b,us-west-2c,us-west-2d"
        },
        {
          "ParameterKey": "DeviceName",
          "ParameterValue": ""
        }
      ],
      "CreationTime": "2021-05-18T18:03:56.292000+00:00",
      "RollbackConfiguration": {},
      "StackStatus": "CREATE_COMPLETE",
      "DisableRollback": false,
      "NotificationARNs": [],
      "Outputs": [
        {
          "OutputKey": "TemplateVersion",
          "OutputValue": "1.1.0",
          "Description": "The version of the template used by Create Cluster Wizard"
        },
        {
          "OutputKey": "UsedByECSCreateCluster",
          "OutputValue": "true",
          "Description": "Flag used by Elastic Container Service Create Cluster Wizard"
        }
      ],
      "Tags": [],
      "DriftInformation": {
        "StackDriftStatus": "NOT_CHECKED"
      }
    }
  ]
}
What am I missing ? Thank you in advance!!
Config screenshots:
k

Kevin Kho

05/19/2021, 3:15 AM
I’m not seeing anything immediate but compare your role against this? https://github.com/PrefectHQ/prefect/pull/4302#issuecomment-814650079
Your agent also needs credentials written here .
upvote 1
2 Views
Powered by