CA Lee

    CA Lee

    1 year ago
    Hello all, trying out the latest Prefect version
    0.14.12
    Running into this error when attempting to run a flow using ECS agent and ECSRun:
    botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.
    I have a working config for prefect agent that executes the flow without errors. However, this involves creating a task-definitions.yaml:
    prefect agent ecs start -t token \
        -n aws-ecs-agent \
        -l label \
        --task-definition /path/to/task-definition.yaml \
        --cluster cluster_arn
    task-definitions.yaml
    networkMode: awsvpc
    cpu: 1024
    memory: 2048
    taskRoleArn: task_role_arn
    executionRoleArn: execution_role_arn
    The flow runs without errors, so the error is not due to IAM permissions. However, when running the ECS Agent using the
    --task-role-arn
    and
    --execution-role-arn
    CLI args, I run into the above-mentioned error. I have also tried running Prefect agent using
    --launch-type FARGATE
    , which I believe is the default and does not need to be specified, but this does not work too.
    prefect agent ecs start -t token \
        -n aws-ecs-agent \
        -l ecs \
        --task-role-arn task_role_arn \
        --execution-role-arn execution_role_arn \
        --cluster cluster_arn
    I have also tried to pass in
    task_role_arn
    and
    execution_role_arn
    into the ECSRun() function within my flow, and ran into the same error. Is there any way to run ECS Agent using CLI args without using the task-definition file?
    In the AWS docs for Task Definition, under Network Mode, it says:
    If you are using the Fargate launch type, the
    awsvpc
    network mode is required.
    However, it does not seem that there is a cli arg to pass to
    prefect agent ecs start
    . I have also tried passing in a
    networkConfiguration
    dict to
    run_task_kwargs
    arg in ECSRun():
    "networkConfiguration": {
        "awsvpcConfiguration": {
            'assignPublicIp': 'ENABLED', 
            'subnets': ['subnet-1', 'subnet-2', 'subnet-3'], 
            'securityGroups': []
        }
    }
    Still running into the same error. Seems that the agent needs to know about
    awsvpc
    as the network mode, but there doesn't seem to be a way to tell it without using a task definitions file
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @CA Lee, thanks for the thorough explanation. I'm going to open an issue for this in the Prefect Core repo as this looks like it may need a PR to address. @Marvin open "ECSRun fails with boto
    InvalidParameterException
    needs awsvpc network mode"
    Marvin

    Marvin

    1 year ago
    Jay Sundaram

    Jay Sundaram

    1 year ago
    @CA Lee, thank you for your post. i encountered the same InvalidParameterException. once i added the --task-definition to the prefect agent ecs start the agent was able to submit the flow for execution. i'll watch issue 4243 for an update.
    CA Lee

    CA Lee

    1 year ago
    Hey @Jay Sundaram @Michael Adkins no worries at all - I’m still experimenting and trying to perfect an ECS flow run with S3 as storage. I noticed in the source code, under src > prefect > agent > ecs , there is a sample
    task-definition.yml
    file provided. I’d like to point out that specifying the below in the yml file leads to the error:
    An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.name should not be null or empty.
    containerDefinitions:
      - name: flow
    I tried verifying by trying to create a dummy task-def using the AWS console, and the generated json file looks like this (doesn’t look all that different from the yaml format above …)
    "containerDefinitions": [
      {
      ...
      "name": "flow"
      }
    ]
    Wondering if there is a yaml-json formatting issue here ? As a side note on another issue which I posted in this channel here, I was running into an error where S3 would not pull the latest image from ECR if I had used
    idempotency_key=serialized_hash()
    to prevent flows from registering again. I was thinking whether adding the following to the
    task-definitions.yml
    would help to force the ECS Fargate container to pull only the latest image:
    containerDefinitions:
      - image: xxx:latest
    Michael Adkins

    Michael Adkins

    1 year ago
    @Jay Sundaram -- could you share your task-definition change that fixed this awsvpc issue?
    @CA Lee -- the json shows a list of container definitions while in your example you're passing a single item without wrapping it in a list
    Jay Sundaram

    Jay Sundaram

    1 year ago
    @Michael Adkins: I basically copied CA Lee's example above:
    networkMode: awsvpc
    cpu: 1024
    memory: 2048
    taskRoleArn: arn:aws:iam::<account id>:role/<ecs-role>
    executionRoleArn: arn:aws:iam::<account id>:role/<ecs-role>
    be sure to replace <account id> with your AWS account id and <ecs-role> with the role you created for executing the tasks pls let me know if i can help further
    Michael Adkins

    Michael Adkins

    1 year ago
    cc @Jim Crist-Harif
    Jay Sundaram

    Jay Sundaram

    1 year ago
    @CA Lee: thanks for your previous post. i have yet to try registering flows using prefect register. i'll follow-up with my findings when i try (soon).
    CA Lee

    CA Lee

    1 year ago
    @Michael Adkins The prefect agent expects the task-def to be in YAML format, so following the syntax guidelines here, All members of a list are lines beginning at the same indentation level starting with a 
    "- "
     (a dash and a space).
    Here is my full task-definitions.yml to be passed to the agent, with the 3 bottom lines passed in as a list according to the syntax guide, which still results in the
    Container.name
    null error when attempting to run a flow using this agent:
    cpu: 1 vCPU
    memory: 2 GB
    networkMode: awsvpc
    taskRoleArn: task_arn
    executionRoleArn: execution_arn
    containerDefinitions: 
      - name: flow
      - image: image_name:latest
    Michael Adkins

    Michael Adkins

    1 year ago
    Can you try removing the
    -
    from the
    image
    line?