https://prefect.io logo
Join Slack
Powered by
Hi, everyone! I'm currently trying to migrate my P...
# ask-community
v
Hi, everyone! I'm currently trying to migrate my Prefect infrastructure to Prefect 2. Currently I have an existing AWS ECS service that boots up a agent and an healthy work pool, and it's waiting for deployments. However, when I try to run one in my machine, I get the following error - 
Submission failed. KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'."
, though I have two ECS blocks registered (one with VPC definition, and other without them). I tried the solution described in this thread - installing 
prefect-aws
in the agent, but to no avail. Does someone has other possible solutions to this issue? I'm adding the ECS task definition, Prefect versions, in the thread. Thank you!
ECS task definition (not exposing secrets and other sensible stuff, replaced with 
---
) - I also log stuff in DataDog for easier debugging.
Copy code
{
  "taskDefinitionArn": "---:task-definition/Prefect2-Production-8vpid:30",
  "containerDefinitions": [
    {
      "name": "app",
      "image": "prefecthq/prefect:2.8-python3.9",
      "cpu": 512,
      "memory": 1024,
      "portMappings": [
        {
          "name": "app-8080-tcp",
          "containerPort": 8080,
          "hostPort": 8080,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "entryPoint": [
        "prefect",
        "agent",
        "start",
        "--pool",
        "default-agent-pool",
        "-q",
        "production"
      ],
      "environment": [
        {
          "name": "EXTRA_PIP_PACKAGES",
          "value": "prefect-aws"
        },
        {
          "name": "env",
          "value": "production"
        },
        {
          "name": "PREFECT_API_KEY",
          "value": "---"
        },
        {
          "name": "PREFECT_API_URL",
          "value": "---"
        }
      ],
      "mountPoints": [],
      "volumesFrom": [],
      "linuxParameters": {
        "initProcessEnabled": true
      },
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Host": "<http://http-intake.logs.datadoghq.eu|http-intake.logs.datadoghq.eu>",
          "Name": "datadog",
          "TLS": "on",
          "compress": "gzip",
          "dd_service": "prefect2",
          "dd_source": "prefect2",
          "dd_tags": "project:prefect,env:production",
          "provider": "ecs"
        },
        "secretOptions": [
          {
            "name": "apikey",
            "valueFrom": "/datadog/apikey"
          }
        ]
      }
    },
    {
      "name": "log_router",
      "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:latest",
      "cpu": 0,
      "portMappings": [],
      "essential": true,
      "environment": [],
      "mountPoints": [],
      "volumesFrom": [],
      "user": "0",
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "enable-ecs-log-metadata": "true"
        }
      }
    }
  ],
  "family": "Prefect2-Production-8vpid",
  "taskRoleArn": "---",
  "executionRoleArn": "---",
  "networkMode": "awsvpc",
  "revision": 30,
  "volumes": [],
  "status": "ACTIVE",
  "requiresAttributes": [
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "name": "ecs.capability.firelens.fluentbit"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
    },
    {
      "name": "com.amazonaws.ecs.capability.logging-driver.awsfirelens"
    },
    {
      "name": "ecs.capability.secrets.ssm.bootstrap.log-driver"
    },
    {
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "name": "ecs.capability.task-eni"
    }
  ],
  "placementConstraints": [],
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "512",
  "memory": "1024",
  "registeredAt": "2023-03-07T18:05:48.635Z",
  "registeredBy": "---",
  "tags": []
}
prefect
version: 
2.8.4
prefect-aws
version: 
0.2.4
I have also ran 
prefect block register -m prefect_aws.ecs
before building deployments, and it tells me it has registered successfully blocks for Credentials and ECS task, though I don't understand if I need to do anything else in the Cloud.
Also, deployment build command, an example: 
prefect deployment build -n test_deploy -q production -ib ecs-task/ecsconfig -a flows/tableau_query_fetch/query_fetch_v2.py:flow
z
Hi! It really sounds like the block implementation is not available on the agent
What happens if you run 
pip show prefect-aws
?
v
Hi @Zanie! Thank you for checking in 🙂 
pip show prefect-aws
returns:
Copy code
Name: prefect-aws
Version: 0.2.4
Summary: Prefect collection of tasks and subflows to integrate with AWS
Home-page: <https://github.com/PrefectHQ/prefect-aws>
Author: Prefect Technologies, Inc.
Author-email: <mailto:help@prefect.io|help@prefect.io>
License: Apache License 2.0
Location: /usr/local/lib/python3.10/site-packages
Requires: boto3, botocore, mypy-boto3-s3, mypy-boto3-secretsmanager, prefect
Required-by:
z
And locally when you run an agent you encounter that error?
What’s the output of 
prefect version
? Is it installed in the same Python environment?
v
Fails to pull the ECR image, though it exists in AWS. 
prefect version == 2.8.4
, same env.
Copy code
15:49:55.602 | INFO    | prefect.agent - Submitting flow run '3c401746-d01a-4625-b847-c468b7dde7b1'
15:49:57.107 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Retrieving task definition 'prefect__tableau_custom_query_fetch__test_deploy'...
15:49:58.602 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Creating task run...
15:49:59.145 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Waiting for task run to start...
15:49:59.214 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is PROVISIONING.
15:50:09.357 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is PENDING.
15:50:24.576 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is DEPROVISIONING.
15:50:34.720 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'neat-potoo': Status is STOPPED.
15:50:34.722 | ERROR   | prefect.agent - Failed to submit flow run '3c401746-d01a-4625-b847-c468b7dde7b1' to infrastructure.
(...)
prefect_aws.ecs.TaskFailedToStart: CannotPullContainerError: pull image manifest has been retried 1 time(s): failed to resolve ref <http://771426788622.dkr.ecr.eu-west-1.amazonaws.com/prefect2-custom:test|771426788622.dkr.ecr.eu-west-1.amazonaws.com/prefect2-custom:test>: pulling from host <http://771426788622.dkr.ecr.eu-west-1.amazonaws.com|771426788622.dkr.ecr.eu-west-1.amazonaws.com> failed with status code [manifests test]: 403 Forbidden
z
Sounds like an authentication issue there
ECR authentication is tricky
v
Very tricky - I haven't tried to authenticate yet after your answer but I'll do it ASAP. However, can you provide some input on how this is related with the first issue? I feel block implementation and ECR login come from two different worlds
z
It shouldn’t be related to the first issue
Here you can see the 
ecs-task
block is retrieved successfully
v
Ah yes, didn't think of it that way. In this case, what could be the diff between local and ECS? I added the 
prefect-aws
to the task definition in ECS, but I'm looking for other possibilities
z
ECS might not respect our entrypoint
(that we set on the image)
So perhaps 
EXTRA_PIP_PACKAGES
is ignored
You may need to build a custom image with prefect-aws installed
Or install it before starting the agent in your agent’s task definition
v
Hi @Zanie just as a follow-up: I created a custom image that installs 
prefect-aws
and registers blocks, it solved the issue. Thank you for helping out!
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by