David Wang
02/02/2022, 4:02 PMessential container in task exited
. The prefect ecs agent also does not appear in the UI. I have been able to start the ECS agent locally, but am unable to start it through aws. Another thing to mention is that when I try to add the logConfiguration to try to see what could be going wrong with the service it will give me an error of ResourceInitializationError: failed to validate logger args: : signal: killed
. I’ve double checked with devOps that the IAM roles and network configurations should be correct too.
Any ideas on how to debug this or why this is happening?Kevin Kho
ResourceInitializationError
myself. Are you using the base Prefect image or your own? Are you just following the docs or do you have any added configuration?David Wang
02/02/2022, 4:06 PMKevin Kho
David Wang
02/02/2022, 4:29 PMKevin Kho
David Wang
02/02/2022, 4:32 PM"image": "prefecthq/prefect:latest-python3.8",
Kevin Kho
David Wang
02/02/2022, 10:20 PM"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "$ECS_LOG_GROUP_NAME",
"awslogs-region": "$AWS_REGION",
"awslogs-stream-prefix": "ecs",
"awslogs-create-group": "true"
}
}
Kevin Kho
task exited
can also be due to a lack of IAM permissionsDavid Wang
02/07/2022, 4:21 PMCannotPullContainerError: inspect image has been retried 5 time(s): failed to resolve ref "<http://docker.io/prefecthq/prefect:latest-python3.8|docker.io/prefecthq/prefect:latest-python3.8>": failed to do request: Head <https://registry-1.docker.io/v2/prefecthq/prefect/manifests/latest-python3.8>: dial tcp ...
error while I was messing around with the security groupsAnna Geller
assignPublicIp=ENABLED
You can set up as part of the network configuration:
aws ecs create-service \
--service-name $ECS_SERVICE_NAME\
--task-definition $ECS_SERVICE_NAME:1 \
--desired-count 1 \
--launch-type FARGATE \
--platform-version LATEST \
--cluster $ECS_CLUSTER_NAME \
--network-configuration awsvpcConfiguration="{subnets=[$SUBNET1, $SUBNET2, $SUBNET3],assignPublicIp=ENABLED}" --region $AWS_REGION
To explain it a bit more - my intuition (not 100% sure) is that your flow run container doesn't have access to the Internet in order to pull the container image from DockerhubDavid Wang
02/07/2022, 6:50 PMAnna Geller
David Wang
02/07/2022, 6:59 PMAnna Geller
David Wang
02/07/2022, 9:06 PMAnna Geller
David Wang
02/08/2022, 4:08 PMResourceInitializationError: failed to validate logger args: : signal: killed
error again. And even if I remove the log configuration part to see if it will run it will give the essential container in task exited
Kevin Kho
David Wang
02/08/2022, 10:33 PMParameter validation failed: Missing required parameter in networkConfiguration.awsvpcConfiguration: "subnets" Unknown parameter in networkConfiguration.awsvpcConfiguration: "Subnets", must be one of: subnets, securityGroups, assignPublicIp
. Do I need to specify network configurations again somewhere in the run config?Kevin Kho
David Wang
02/09/2022, 4:19 PMKevin Kho
David Wang
02/09/2022, 4:24 PMKevin Kho
task_definition_path
. That links to here , and then you can specify those subnet and security group there.
The ECS agent also takes a definition upon starting . You can pass --task_definition_path
. You just need to make sure that these live somewhere the agent can pull during runtime (like an S3 bucket it has access to).
The agent task_definition serves as a default for the Flows that it runs, but the RunConfig can override it. The default one the agent uses can be found hereDavid Wang
02/09/2022, 8:10 PMnetworkMode: awsvpc
cpu: 512
memory: 1024
containerDefinitions:
- name: prefectEcsAgent
networkConfiguration:
awsvpcConfiguration:
Subnets:
- subnet-xxx
securityGroups:
- sg-xxx
assignPublicIp: DISABLED
Kevin Kho
run_task_kwargs
or on the agent
If you specify the awsvpc network mode, the task is allocated an elastic network interface, and you must specify a NetworkConfiguration when you create a service or run a task with the task definition.
This is so painful I dunno why you can’t do it as part of the task definitionDavid Wang
02/09/2022, 10:17 PMKevin Kho
David Wang
02/10/2022, 3:57 PM{
"family": "$ECS_SERVICE_NAME",
"requiresCompatibilities": [
"FARGATE"
],
"networkMode": "awsvpc",
"cpu": "512",
"memory": "1024",
"taskRoleArn": "arn:aws:iam::xxx:role/prefectTaskRole",
"executionRoleArn": "arn:aws:iam::xxx:role/prefectECSAgentTaskExecutionRole",
"containerDefinitions": [
{
"name": "$ECS_SERVICE_NAME",
"image": "prefecthq/prefect",
"essential": true,
"command": [
"prefect",
"agent",
"ecs",
"start",
"--run-task-kwargs",
"<s3://xxx-test-bucket/david/ecs-config.yaml>"
],
"environment": [
{
"name": "PREFECT__CLOUD__API_KEY",
"value": "xxx"
},
{
"name": "PREFECT__CLOUD__AGENT__LABELS",
"value": "['dev']"
},
{
"name": "PREFECT__CLOUD__AGENT__LEVEL",
"value": "INFO"
},
{
"name": "PREFECT__CLOUD__API",
"value": "<https://api.prefect.io>"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "$ECS_LOG_GROUP_NAME",
"awslogs-region": "$AWS_REGION",
"awslogs-stream-prefix": "ecs",
"awslogs-create-group": "true"
}
}
}
]
}
and inside the yaml file
networkConfiguration:
awsvpcConfiguration:
Subnets:
- subnet-xxx
securityGroups:
- sg-xxx
assignPublicIp: DISABLED
I also have the network configuration setup when creating the service with aws ecs create-service
aws ecs create-service \
--service-name $ECS_SERVICE_NAME\
--task-definition $ECS_SERVICE_NAME:1 \
--desired-count 1 \
--launch-type FARGATE \
--cluster $ECS_CLUSTER_NAME \
--network-configuration awsvpcConfiguration="{subnets=[$SUBNET1],securityGroups=[$SECURITYGROUP]}" --region $AWS_REGION
RUN_CONFIG = ECSRun(
labels=["dev"],
task_role_arn="arn:aws:iam::xxx:role/prefectTaskRole",
run_task_kwargs=dict(cluster="prefectEcsCluster",networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
'subnet-xxx'
],
'securityGroups': [
'sg-xxx',
],
'assignPublicIp': 'DISABLED'
}
}),
)
Christopher
02/11/2022, 6:53 AMDavid Wang
02/11/2022, 4:28 PM