https://prefect.io logo
Powered by
Title
v

Vasco Leitão

03/08/2023, 11:23 AM
Hi, everyone! I'm currently trying to migrate my Prefect infrastructure to Prefect 2. Currently I have an existing AWS ECS service that boots up a agent and an healthy work pool, and it's waiting for deployments. However, when I try to run one in my machine, I get the following error - 
Submission failed. KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'."
, though I have two ECS blocks registered (one with VPC definition, and other without them). I tried the solution described in this thread - installing 
prefect-aws
in the agent, but to no avail. Does someone has other possible solutions to this issue? I'm adding the ECS task definition, Prefect versions, in the thread. Thank you!
ECS task definition (not exposing secrets and other sensible stuff, replaced with 
---
) - I also log stuff in DataDog for easier debugging. 
{
  "taskDefinitionArn": "---:task-definition/Prefect2-Production-8vpid:30",
  "containerDefinitions": [
    {
      "name": "app",
      "image": "prefecthq/prefect:2.8-python3.9",
      "cpu": 512,
      "memory": 1024,
      "portMappings": [
        {
          "name": "app-8080-tcp",
          "containerPort": 8080,
          "hostPort": 8080,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "entryPoint": [
        "prefect",
        "agent",
        "start",
        "--pool",
        "default-agent-pool",
        "-q",
        "production"
      ],
      "environment": [
        {
          "name": "EXTRA_PIP_PACKAGES",
          "value": "prefect-aws"
        },
        {
          "name": "env",
          "value": "production"
        },
        {
          "name": "PREFECT_API_KEY",
          "value": "---"
        },
        {
          "name": "PREFECT_API_URL",
          "value": "---"
        }
      ],
      "mountPoints": [],
      "volumesFrom": [],
      "linuxParameters": {
        "initProcessEnabled": true
      },
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Host": "<http://http-intake.logs.datadoghq.eu|http-intake.logs.datadoghq.eu>",
          "Name": "datadog",
          "TLS": "on",
          "compress": "gzip",
          "dd_service": "prefect2",
          "dd_source": "prefect2",
          "dd_tags": "project:prefect,env:production",
          "provider": "ecs"
        },
        "secretOptions": [
          {
            "name": "apikey",
            "valueFrom": "/datadog/apikey"
          }
        ]
      }
    },
    {
      "name": "log_router",
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:latest",
      "cpu": 0,
      "portMappings": [],
      "essential": true,
      "environment": [],
      "mountPoints": [],
      "volumesFrom": [],
      "user": "0",
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "enable-ecs-log-metadata": "true"
        }
      }
    }
  ],
  "family": "Prefect2-Production-8vpid",
  "taskRoleArn": "---",
  "executionRoleArn": "---",
  "networkMode": "awsvpc",
  "revision": 30,
  "volumes": [],
  "status": "ACTIVE",
  "requiresAttributes": [
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "name": "ecs.capability.firelens.fluentbit"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
    },
    {
      "name": "com.amazonaws.ecs.capability.logging-driver.awsfirelens"
    },
    {
      "name": "ecs.capability.secrets.ssm.bootstrap.log-driver"
    },
    {
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "name": "ecs.capability.task-eni"
    }
  ],
  "placementConstraints": [],
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "512",
  "memory": "1024",
  "registeredAt": "2023-03-07T18:05:48.635Z",
  "registeredBy": "---",
  "tags": []
}
prefect
version: 
2.8.4
prefect-aws
version: 
0.2.4
I have also ran 
prefect block register -m prefect_aws.ecs
before building deployments, and it tells me it has registered successfully blocks for Credentials and ECS task, though I don't understand if I need to do anything else in the Cloud.
Also, deployment build command, an example: 
prefect deployment build -n test_deploy -q production -ib ecs-task/ecsconfig -a flows/tableau_query_fetch/query_fetch_v2.py:flow
z

Zanie

03/09/2023, 3:34 PM
Hi! It really sounds like the block implementation is not available on the agent
What happens if you run 
pip show prefect-aws
?
v

Vasco Leitão

03/09/2023, 3:36 PM
Hi @Zanie! Thank you for checking in 🙂 
pip show prefect-aws
returns: 
Name: prefect-aws
Version: 0.2.4
Summary: Prefect collection of tasks and subflows to integrate with AWS
Home-page: <https://github.com/PrefectHQ/prefect-aws>
Author: Prefect Technologies, Inc.
Author-email: <mailto:help@prefect.io|help@prefect.io>
License: Apache License 2.0
Location: /usr/local/lib/python3.10/site-packages
Requires: boto3, botocore, mypy-boto3-s3, mypy-boto3-secretsmanager, prefect
Required-by:
z

Zanie

03/09/2023, 3:38 PM
And locally when you run an agent you encounter that error?
What’s the output of 
prefect version
? Is it installed in the same Python environment?
v

Vasco Leitão

03/09/2023, 3:52 PM
Fails to pull the ECR image, though it exists in AWS. 
prefect version == 2.8.4
, same env. 
15:49:55.602 | INFO    | prefect.agent - Submitting flow run '3c401746-d01a-4625-b847-c468b7dde7b1'
15:49:57.107 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Retrieving task definition 'prefect__tableau_custom_query_fetch__test_deploy'...
15:49:58.602 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Creating task run...
15:49:59.145 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Waiting for task run to start...
15:49:59.214 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is PROVISIONING.
15:50:09.357 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is PENDING.
15:50:24.576 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is DEPROVISIONING.
15:50:34.720 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is STOPPED.
15:50:34.722 | ERROR   | prefect.agent - Failed to submit flow run '3c401746-d01a-4625-b847-c468b7dde7b1' to infrastructure.
(...)
prefect_aws.ecs.TaskFailedToStart: CannotPullContainerError: pull image manifest has been retried 1 time(s): failed to resolve ref <http://771426788622.dkr.ecr.eu-west-1.amazonaws.com/prefect2-custom:test|771426788622.dkr.ecr.eu-west-1.amazonaws.com/prefect2-custom:test>: pulling from host <http://771426788622.dkr.ecr.eu-west-1.amazonaws.com|771426788622.dkr.ecr.eu-west-1.amazonaws.com> failed with status code [manifests test]: 403 Forbidden
z

Zanie

03/09/2023, 3:57 PM
Sounds like an authentication issue there
ECR authentication is tricky
v

Vasco Leitão

03/09/2023, 5:10 PM
Very tricky - I haven't tried to authenticate yet after your answer but I'll do it ASAP. However, can you provide some input on how this is related with the first issue? I feel block implementation and ECR login come from two different worlds
z

Zanie

03/09/2023, 5:13 PM
It shouldn’t be related to the first issue
Here you can see the 
ecs-task
block is retrieved successfully
v

Vasco Leitão

03/09/2023, 5:22 PM
Ah yes, didn't think of it that way. In this case, what could be the diff between local and ECS? I added the 
prefect-aws
to the task definition in ECS, but I'm looking for other possibilities
z

Zanie

03/09/2023, 5:22 PM
ECS might not respect our entrypoint
(that we set on the image)
So perhaps 
EXTRA_PIP_PACKAGES
is ignored
You may need to build a custom image with prefect-aws installed
Or install it before starting the agent in your agent’s task definition
v

Vasco Leitão

03/14/2023, 10:23 AM
Hi @Zanie just as a follow-up: I created a custom image that installs 
prefect-aws
and registers blocks, it solved the issue. Thank you for helping out!
🙌 1
#prefect-communityJoin Slack
Powered by