Yaron Levi
04/26/2023, 6:35 PMprefect deployment build jobs/test1/test1.py:test1 \
--name test1 \
--output jobs/test1/deployment \
--storage-block github/yuvital-prefect-github \
--pool test-pool \
--infra-block ecs-task/ecs-test-block
Meaning when the Work Pool Task poll for a new flow to run, it should also create a new ECS Task for that flow run.
Looking at the logs I can see that the work pool do get the new flow to run, but fails with:
Submission failed. KeyError: āNo class found for dispatch key āecs-taskā in registry for type āBlockā.āZanie
04/26/2023, 6:38 PMprefect-aws
isnāt installed in your workerās environment ā but note that workers are meant to be used with the ECSWorker
not the ECSTask
blockecs
rather than prefect-agent
Yaron Levi
04/26/2023, 6:39 PM{
"taskDefinitionArn": "arn:aws:ecs:eu-central-1:759317626178:task-definition/agent-task:8",
"containerDefinitions": [
{
"name": "agent-task",
"image": "prefecthq/prefect:2-python3.10",
"cpu": 0,
"portMappings": [
{
"name": "agent-task-80-tcp",
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp",
"appProtocol": "http"
}
],
"essential": true,
"command": [
"prefect",
"worker",
"start",
"--pool",
"test-pool"
],
"environment": [
{
"name": "PREFECT_API_KEY",
"value": ""
},
{
"name": "PREFECT_API_URL",
"value": ""
},
{
"name": "EXTRA_PIP_PACKAGES",
"value": "prefect-aws s3fs"
}
],
"environmentFiles": [],
"mountPoints": [],
"volumesFrom": [],
"ulimits": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/agent-task",
"awslogs-region": "eu-central-1",
"awslogs-stream-prefix": "ecs"
}
}
}
],
"family": "agent-task",
"executionRoleArn": "arn:aws:iam::759317626178:role/ecsTaskExecutionRole",
"networkMode": "awsvpc",
"revision": 8,
"volumes": [],
"status": "ACTIVE",
"requiresAttributes": [
{
"name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
},
{
"name": "ecs.capability.execution-role-awslogs"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
},
{
"name": "ecs.capability.task-eni"
},
{
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
}
],
"placementConstraints": [],
"compatibilities": [
"EC2",
"FARGATE"
],
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024",
"memory": "3072",
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
},
"registeredAt": "2023-04-26T17:33:57.713Z",
"registeredBy": "arn:aws:iam::759317626178:root",
"tags": []
}
Zanie
04/26/2023, 6:40 PMEXTRA_PIP_PACKAGES
because it bypasses our entrypointYaron Levi
04/26/2023, 6:40 PMZanie
04/26/2023, 6:41 PM/opt/prefect/entrypoint.sh
at the start of your commands itāll install prefect-aws
like you wantYaron Levi
04/26/2023, 6:41 PMZanie
04/26/2023, 6:42 PMYaron Levi
04/26/2023, 6:42 PMZanie
04/26/2023, 6:44 PMYaron Levi
04/26/2023, 6:45 PMZanie
04/26/2023, 6:45 PMYaron Levi
04/26/2023, 6:46 PMprefect deployment build jobs/test1/test1.py:test1 \
--name test1 \
--output jobs/test1/deployment \
--storage-block github/yuvital-prefect-github \
--pool test-pool \
--infra-block ecs-task/ecs-test-block
I should do:
prefect deployment build jobs/test1/test1.py:test1 \
--name test1 \
--output jobs/test1/deployment \
--storage-block github/yuvital-prefect-github \
--pool test-pool
alex
04/26/2023, 6:47 PMecs
work pool, youāll want to use the new Prefect Projects functionality. Following the project tutorial is a good way to get started: https://docs.prefect.io/latest/tutorials/projects/Yaron Levi
04/26/2023, 6:48 PMalex
04/26/2023, 6:51 PMYaron Levi
04/26/2023, 6:52 PMalex
04/26/2023, 6:57 PMYaron Levi
04/26/2023, 6:59 PM