Hello all trying out the latest Prefect version 0 14 12 Runn | Prefect-community

botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.

I have a working config for prefect agent that executes the flow without errors. However, this involves creating a task-definitions.yaml:

prefect agent ecs start -t token \
    -n aws-ecs-agent \
    -l label \
    --task-definition /path/to/task-definition.yaml \
    --cluster cluster_arn

task-definitions.yaml

networkMode: awsvpc
cpu: 1024
memory: 2048
taskRoleArn: task_role_arn
executionRoleArn: execution_role_arn

The flow runs without errors, so the error is not due to IAM permissions. However, when running the ECS Agent using the

--task-role-arn

and

--execution-role-arn

CLI args, I run into the above-mentioned error. I have also tried running Prefect agent using

--launch-type FARGATE

, which I believe is the default and does not need to be specified, but this does not work too.

prefect agent ecs start -t token \
    -n aws-ecs-agent \
    -l ecs \
    --task-role-arn task_role_arn \
    --execution-role-arn execution_role_arn \
    --cluster cluster_arn

I have also tried to pass in

task_role_arn

and

execution_role_arn

into the ECSRun() function within my flow, and ran into the same error. Is there any way to run ECS Agent using CLI args without using the task-definition file?

In the AWS docs for Task Definition, under Network Mode, it says:

If you are using the Fargate launch type, the
awsvpc
network mode is required.

However, it does not seem that there is a cli arg to pass to

prefect agent ecs start

. I have also tried passing in a

networkConfiguration

dict to

run_task_kwargs

arg in ECSRun():

"networkConfiguration": {
    "awsvpcConfiguration": {
        'assignPublicIp': 'ENABLED', 
        'subnets': ['subnet-1', 'subnet-2', 'subnet-3'], 
        'securityGroups': []
    }
}

Still running into the same error. Seems that the agent needs to know about

awsvpc

as the network mode, but there doesn't seem to be a way to tell it without using a task definitions file

Michael Adkins

1 year ago

Hey @CA Lee, thanks for the thorough explanation. I'm going to open an issue for this in the Prefect Core repo as this looks like it may need a PR to address. @Marvin open "ECSRun fails with boto

InvalidParameterException

needs awsvpc network mode"

Marvin

1 year ago

https://github.com/PrefectHQ/prefect/issues/4243

Jay Sundaram

1 year ago

@CA Lee, thank you for your post. i encountered the same InvalidParameterException. once i added the --task-definition to the prefect agent ecs start the agent was able to submit the flow for execution. i'll watch issue 4243 for an update.

CA Lee

1 year ago

Hey @Jay Sundaram @Michael Adkins no worries at all - I’m still experimenting and trying to perfect an ECS flow run with S3 as storage. I noticed in the source code, under src > prefect > agent > ecs , there is a sample

task-definition.yml

file provided. I’d like to point out that specifying the below in the yml file leads to the error:

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.name should not be null or empty.

containerDefinitions:
  - name: flow

I tried verifying by trying to create a dummy task-def using the AWS console, and the generated json file looks like this (doesn’t look all that different from the yaml format above …)

"containerDefinitions": [
  {
  ...
  "name": "flow"
  }
]

Wondering if there is a yaml-json formatting issue here ? As a side note on another issue which I posted in this channel here, I was running into an error where S3 would not pull the latest image from ECR if I had used

idempotency_key=serialized_hash()

to prevent flows from registering again. I was thinking whether adding the following to the

task-definitions.yml

would help to force the ECS Fargate container to pull only the latest image:

containerDefinitions:
  - image: xxx:latest

Michael Adkins

1 year ago

@Jay Sundaram -- could you share your task-definition change that fixed this awsvpc issue?

@CA Lee -- the json shows a list of container definitions while in your example you're passing a single item without wrapping it in a list

Jay Sundaram

1 year ago

@Michael Adkins: I basically copied CA Lee's example above:

networkMode: awsvpc
cpu: 1024
memory: 2048
taskRoleArn: arn:aws:iam::<account id>:role/<ecs-role>
executionRoleArn: arn:aws:iam::<account id>:role/<ecs-role>

be sure to replace <account id> with your AWS account id and <ecs-role> with the role you created for executing the tasks pls let me know if i can help further

Michael Adkins

1 year ago

cc @Jim Crist-Harif

Jay Sundaram

1 year ago

@CA Lee: thanks for your previous post. i have yet to try registering flows using prefect register. i'll follow-up with my findings when i try (soon).

CA Lee

1 year ago

@Michael Adkins The prefect agent expects the task-def to be in YAML format, so following the syntax guidelines here, All members of a list are lines beginning at the same indentation level starting with a
"- "
(a dash and a space). Here is my full task-definitions.yml to be passed to the agent, with the 3 bottom lines passed in as a list according to the syntax guide, which still results in the

Container.name

null error when attempting to run a flow using this agent:

cpu: 1 vCPU
memory: 2 GB
networkMode: awsvpc
taskRoleArn: task_arn
executionRoleArn: execution_arn
containerDefinitions: 
  - name: flow
  - image: image_name:latest

Michael Adkins

1 year ago

Can you try removing the

from the

image

line?

View count: 1

Join thread in Slack