Thread
#prefect-community
    n

    Nikola Lusic

    1 year ago
    Hey guys, so I'm setting up the Prefect environment that will be running tasks on AWS ECS via ECSRun configuration. Everything is working except for one parameter: setting the
    cpu
    and
    memory
    parameters of the ECSRun doesn't seem to have any effect. It seems that the agent always creates a task with 1CPU and 2GB RAM. Do you maybe have some pointers for this issue?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Nikola Lusic, could you show how you’re setting them?
    n

    Nikola Lusic

    1 year ago
    run_config = ECSRun(
        image=ec.get("DOCKER_IMAGE"),
        env=env,
        cpu=512,
        memory=2048)
    storage = S3(bucket=ec.get("S3_BUCKET"))
    result = S3Result(bucket=ec.get("S3_BUCKET"))
    with Flow(
            name="flow_name",
            run_config=run_config,
            storage=storage,
            result=result
    ) as flow:
        flow_steps()
    As I've mentioned, the flow works, and the logic is being executed, but the ECS task CPU and memory are always 1vcpu/2GB RAM, no matter what parameters I pass to ECSRun.
    Kevin Kho

    Kevin Kho

    1 year ago
    Have you tried something like?
    flow.run_config = ECSRun(cpu="2 vcpu", memory="4 GB")
    Actually what you have should work. Let me take a look some more
    n

    Nikola Lusic

    1 year ago
    I have tried both int values and the string "2 vcpu"/"4 GB"
    Kevin Kho

    Kevin Kho

    1 year ago
    You’re not supplying a task definition right?
    n

    Nikola Lusic

    1 year ago
    nope
    Kevin Kho

    Kevin Kho

    1 year ago
    I’m digging through the agent code.
    n

    Nikola Lusic

    1 year ago
    I appreciate it. I'm combing through the docs, but there are not too many mentions of the CPU/RAM parameters for ECS tasks.
    @Kevin Kho the agent that I've deployed has no special configuration, other than the connection to Prefect Sever.
    Kevin Kho

    Kevin Kho

    1 year ago
    Yeah I’ll get back to this in a bit. Just have a meeting first.
    n

    Nikola Lusic

    1 year ago
    Just an update on my testing: If I define illegal cpu/ram arguments for an ECS run (for example, 256/256), the ECS task fails, altough, on ECS task definition it's again defined as 1CPU/2GB RAM task.
    Kevin Kho

    Kevin Kho

    1 year ago
    What version of Prefect are you on?
    n

    Nikola Lusic

    1 year ago
    0.14.17
    Michael Adkins

    Michael Adkins

    1 year ago
    Hey @Nikola Lusic -- just to clarify, are you trying to set the CPU and memory for the flow container in your ECS task?
    n

    Nikola Lusic

    1 year ago
    Indeed. We have tasks that require variable amounts of resources, but no matter when parameters of
    cpu
    and
    memory
    I set in the
    ECSRun
    , the result is always an ECS task that has 1 CPU and 2 GB RAM.
    Michael Adkins

    Michael Adkins

    1 year ago
    It looks like it can be set at the container level and at the task definition level and we're only overriding the task definition value
    Would you be able to share a redacted task definition from one of your runs?
    n

    Nikola Lusic

    1 year ago
    This is the task definition created by Prefect that is running the ECS container:
    {
      "ipcMode": null,
      "executionRoleArn": "<AWS ROLE>",
      "containerDefinitions": [
        {
          "dnsSearchDomains": null,
          "environmentFiles": null,
          "logConfiguration": null,
          "entryPoint": null,
          "portMappings": [],
          "command": null,
          "linuxParameters": null,
          "cpu": 0,
          "environment": [
            {
              "name": "PREFECT__CONTEXT__IMAGE",
              "value": "<DOCKER IMAGE>""
            }
          ],
          "resourceRequirements": null,
          "ulimits": null,
          "dnsServers": null,
          "mountPoints": [],
          "workingDirectory": null,
          "secrets": null,
          "dockerSecurityOptions": null,
          "memory": null,
          "memoryReservation": null,
          "volumesFrom": [],
          "stopTimeout": null,
          "image": "<DOCKER IMAGE>",
          "startTimeout": null,
          "firelensConfiguration": null,
          "dependsOn": null,
          "disableNetworking": null,
          "interactive": null,
          "healthCheck": null,
          "essential": true,
          "links": null,
          "hostname": null,
          "extraHosts": null,
          "pseudoTerminal": null,
          "user": null,
          "readonlyRootFilesystem": null,
          "dockerLabels": null,
          "systemControls": null,
          "privileged": null,
          "name": "flow"
        }
      ],
      "placementConstraints": [],
      "memory": "2048",
      "taskRoleArn": null,
      "compatibilities": [
        "EC2",
        "FARGATE"
      ],
      "taskDefinitionArn": "<TASK DEFINITION ARN>",
      "family": "prefect-opal-copy",
      "requiresAttributes": [
        {
          "targetId": null,
          "targetType": null,
          "value": null,
          "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
          "targetId": null,
          "targetType": null,
          "value": null,
          "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
          "targetId": null,
          "targetType": null,
          "value": null,
          "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
          "targetId": null,
          "targetType": null,
          "value": null,
          "name": "ecs.capability.task-eni"
        }
      ],
      "pidMode": null,
      "requiresCompatibilities": [
        "FARGATE"
      ],
      "networkMode": "awsvpc",
      "cpu": "1024",
      "revision": 22,
      "status": "INACTIVE",
      "inferenceAccelerators": null,
      "proxyConfiguration": null,
      "volumes": [],
      "statusString": "(INACTIVE)"
    }
    This task definition is the product of the following
    ECSRun
    configuration:
    ECSRun(labels=["<LABEL>"],
           image=ec.get("DOCKER_IMAGE"),
           env=env,
           cpu=2048,
           memory=4096)
    Michael Adkins

    Michael Adkins

    1 year ago
    So I'm not entirely sure how this is reflected in AWS -- but we are passing the CPU and memory as "Overrides" for the specific task run, not in the task definition
    Can you confirm that your actual task run does not have the correct CPU/memory allocated?
    n

    Nikola Lusic

    1 year ago
    I can check, although I am not sure that a ECS container could have more resources allocated to it that it's task definition specifies.
    Michael Adkins

    Michael Adkins

    1 year ago
    We are • Registering a task definition • Creating a task with overrides for some fields of the task definition
    The CPU/memory overrides are for the task level resources, not the container resources
    n

    Nikola Lusic

    1 year ago
    Alright. I'm not sure how to check a Fargate task allocated resources (I can't SSH into the container), and the AWS doesn't offer obvious source of this information. Do you maybe know how to check this?
    Michael Adkins

    Michael Adkins

    1 year ago
    I'll ping someone who's worked with ECS recently on our team.
    Hey @Nikola Lusic -- we have confirmed that the CPU and memory are passed correctly to your task run. Since you define your own task definition we must pass things like this as overrides per run rather than on the task definition. You can use
    aws ecs describe-tasks
    to take a look at your task and see that the settings are correct https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html
    @Marvin archive "ECSRun memory and CPU settings are not set in the task definition"
    Marvin

    Marvin

    1 year ago