Emma Willemsma
10/01/2020, 8:15 PMAn error occurred (InvalidParameterException) when calling the RegisterTaskDefinition operation: Invalid 'cpu' setting for task.
The entrypoint we're using for the agent is:
["prefect","agent","start","fargate","cpu=256","memory=1024","networkConfiguration=$NETWORK_CONFIGURATION"],
Does anyone know what we should we be setting for cpu
to get it working?nicholas
10/01/2020, 8:17 PMcpu="256"
memory
Emma Willemsma
10/01/2020, 8:20 PMnicholas
10/01/2020, 8:25 PMEmma Willemsma
10/02/2020, 2:36 PMcpu
parameter in the entryPoint
and we keep getting the same error. So this for example isn't working:
["prefect","agent","start","fargate","cpu=\"256\"","memory=\"1024\"","networkConfiguration=$NETWORK_CONFIGURATION"],
Has anyone gotten this to work?Dylan
10/02/2020, 2:43 PMEmma Willemsma
10/02/2020, 2:45 PMjosh
10/02/2020, 2:46 PM"cpu='256'"
?Spencer
10/02/2020, 2:47 PM["prefect", "agent", "start", "fargate", "enable_task_revisions=true"]
With the environment set of:
PREFECT__BACKEND: server
PREFECT__CLOUD__AGENT__AGENT_ADDRESS: <http://127.0.0.1:8080>
PREFECT__CLOUD__AGENT__LABELS: '["s3-flow-storage"]'
executionRoleArn: ecs-task-execution-role
memory: 512
cpu: 256
networkConfiguration: ${NETWORK_CONFIGURATION}
taskRoleArn: prefect-agent-role
containerDefinitions_logConfiguration: ${LOG_CONFIGURATION}
cluster: ${CLUSTER_NAME}
Emma Willemsma
10/02/2020, 2:49 PMjosh
10/02/2020, 2:53 PMSpencer
10/02/2020, 2:55 PM_parse_kwargs
stuff is rather difficult to follow 😓 I didn't make a PR because I couldn't quite understand the reason for the complexity (nor had the time to understand the flow).josh
10/02/2020, 2:56 PMEmma Willemsma
10/02/2020, 6:19 PMZaid Naji
10/02/2020, 7:18 PMSpencer
10/02/2020, 7:29 PMFargateTaskEnvironment
that you attach to the flowsZaid Naji
10/02/2020, 7:29 PMSpencer
10/02/2020, 7:30 PMprefect.Flow
) and update their environments to the proper FargateTaskEnvironment
before registering them. Also, I am using the S3Storage
alongside this.
You can of course just set it directly on your flows; I just wasn't a fan of that configuration. I wanted my data engineers to not have to be aware of it.Zaid Naji
10/02/2020, 7:34 PMSpencer
10/02/2020, 7:35 PMZaid Naji
10/02/2020, 7:36 PMSpencer
10/02/2020, 7:37 PMZaid Naji
10/02/2020, 8:16 PMSpencer
10/02/2020, 8:41 PMFargateTaskEnvironment
constructor: taskRoleArn
and executionRoleArn
FargateTaskEnvironment(
# Task Definition
family=task_definition_name,
taskRoleArn=settings.task_role_arn,
executionRoleArn=settings.execution_role_arn,
cpu=settings.cpu,
memory=settings.memory,
containerDefinitions=[
{
"name": "flow-container",
"image": "image",
"command": [],
"environment": [],
"essential": True,
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": settings.awslogs_group,
"awslogs-region": settings.awslogs_region,
"awslogs-stream-prefix": settings.awslogs_stream_prefix,
},
},
}
],
networkMode="awsvpc",
requiresCompatibilities=["FARGATE"],
# Task Run
cluster=settings.cluster_name,
region=aws_settings.region,
taskDefinition=task_definition_name,
launch_type="FARGATE",
networkConfiguration={
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED"
if settings.assign_public_ip
else "DISABLED",
"subnets": settings.subnets,
}
},
metadata={"image": image},
)
Zaid Naji
10/03/2020, 4:31 PMSpencer
10/03/2020, 5:13 PMFargateTaskEnvironment
get overridden by the agent when constructing the task IIRCZaid Naji
10/03/2020, 5:41 PMSpencer
10/03/2020, 6:01 PMFargateTaskEnvironment
with different taskRoleArn
for each role.Zaid Naji
10/03/2020, 6:24 PMSpencer
10/03/2020, 7:11 PMsts:AssumeRole
that the tasks can use directly; then you setup boto3 with an AssumeRoleCredentialsProvider
(or some such; I know boto3 doesn't have this natively but other AWS SDKs do; for boto3 https://stackoverflow.com/a/45834847) in the session?Zaid Naji
10/03/2020, 7:49 PMale
10/05/2020, 1:38 PMSpencer
10/05/2020, 1:39 PMmetadata={"image": ...}
Specifying the taskRoleArn and executionRoleArn are native fields in the FargateTaskEnvironment separate from the metadata field.ale
10/05/2020, 1:40 PM