Hey guys so I m setting up the Prefect environment that will Prefect Community #ask-community

Hey guys, so I'm setting up the Prefect environmen...

Nikola Lusic

05/27/2021, 5:17 PM

Hey guys, so I'm setting up the Prefect environment that will be running tasks on AWS ECS via ECSRun configuration. Everything is working except for one parameter: setting the

cpu

and

memory

parameters of the ECSRun doesn't seem to have any effect. It seems that the agent always creates a task with 1CPU and 2GB RAM. Do you maybe have some pointers for this issue?

Kevin Kho

05/27/2021, 5:18 PM

Hi @Nikola Lusic, could you show how you’re setting them?

Nikola Lusic

05/27/2021, 5:22 PM

Copy code

run_config = ECSRun(
    image=ec.get("DOCKER_IMAGE"),
    env=env,
    cpu=512,
    memory=2048)
storage = S3(bucket=ec.get("S3_BUCKET"))
result = S3Result(bucket=ec.get("S3_BUCKET"))
with Flow(
        name="flow_name",
        run_config=run_config,
        storage=storage,
        result=result
) as flow:
    flow_steps()

Nikola Lusic

05/27/2021, 5:23 PM

As I've mentioned, the flow works, and the logic is being executed, but the ECS task CPU and memory are always 1vcpu/2GB RAM, no matter what parameters I pass to ECSRun.

Kevin Kho

05/27/2021, 5:23 PM

Have you tried something like?

Copy code

flow.run_config = ECSRun(cpu="2 vcpu", memory="4 GB")

Kevin Kho

05/27/2021, 5:25 PM

Actually what you have should work. Let me take a look some more

Nikola Lusic

05/27/2021, 5:26 PM

I have tried both int values and the string "2 vcpu"/"4 GB"

Kevin Kho

05/27/2021, 5:27 PM

You’re not supplying a task definition right?

Nikola Lusic

05/27/2021, 5:27 PM

nope

Kevin Kho

05/27/2021, 5:36 PM

I’m digging through the agent code.

Nikola Lusic

05/27/2021, 5:37 PM

I appreciate it. I'm combing through the docs, but there are not too many mentions of the CPU/RAM parameters for ECS tasks.

Nikola Lusic

05/27/2021, 6:02 PM

@Kevin Kho the agent that I've deployed has no special configuration, other than the connection to Prefect Sever.

Kevin Kho

05/27/2021, 6:03 PM

Yeah I’ll get back to this in a bit. Just have a meeting first.

🙌 1

Nikola Lusic

05/27/2021, 6:29 PM

Just an update on my testing: If I define illegal cpu/ram arguments for an ECS run (for example, 256/256), the ECS task fails, altough, on ECS task definition it's again defined as 1CPU/2GB RAM task.

Kevin Kho

05/27/2021, 7:06 PM

What version of Prefect are you on?

Nikola Lusic

05/27/2021, 7:07 PM

0.14.17

Zanie

05/27/2021, 7:10 PM

Hey @Nikola Lusic -- just to clarify, are you trying to set the CPU and memory for the flow container in your ECS task?

Nikola Lusic

05/27/2021, 7:12 PM

Indeed. We have tasks that require variable amounts of resources, but no matter when parameters of

cpu

and

memory

I set in the

ECSRun

, the result is always an ECS task that has 1 CPU and 2 GB RAM.

Zanie

05/27/2021, 7:12 PM

It looks like it can be set at the container level and at the task definition level and we're only overriding the task definition value

Zanie

05/27/2021, 7:13 PM

Would you be able to share a redacted task definition from one of your runs?

Nikola Lusic

05/27/2021, 7:17 PM

This is the task definition created by Prefect that is running the ECS container:

Copy code

{
  "ipcMode": null,
  "executionRoleArn": "<AWS ROLE>",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": null,
      "entryPoint": null,
      "portMappings": [],
      "command": null,
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "PREFECT__CONTEXT__IMAGE",
          "value": "<DOCKER IMAGE>""
        }
      ],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "<DOCKER IMAGE>",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "flow"
    }
  ],
  "placementConstraints": [],
  "memory": "2048",
  "taskRoleArn": null,
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "<TASK DEFINITION ARN>",
  "family": "prefect-opal-copy",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-ecr-pull"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "1024",
  "revision": 22,
  "status": "INACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": [],
  "statusString": "(INACTIVE)"
}

Nikola Lusic

05/27/2021, 7:18 PM

This task definition is the product of the following

ECSRun

configuration:

Copy code

ECSRun(labels=["<LABEL>"],
       image=ec.get("DOCKER_IMAGE"),
       env=env,
       cpu=2048,
       memory=4096)

Zanie

05/27/2021, 7:19 PM

So I'm not entirely sure how this is reflected in AWS -- but we are passing the CPU and memory as "Overrides" for the specific task run, not in the task definition

Zanie

05/27/2021, 7:20 PM

ie we are using https://boto3.amazonaws.com/v1/documentation/api/1.9.88/reference/services/ecs.html#ECS.Client.run_task with an overrides dictionary

Zanie

05/27/2021, 7:21 PM

Can you confirm that your actual task run does not have the correct CPU/memory allocated?

Nikola Lusic

05/27/2021, 7:24 PM

I can check, although I am not sure that a ECS container could have more resources allocated to it that it's task definition specifies.

Zanie

05/27/2021, 7:25 PM

We are • Registering a task definition • Creating a task with overrides for some fields of the task definition

Zanie

05/27/2021, 7:25 PM

The CPU/memory overrides are for the task level resources, not the container resources

Nikola Lusic

05/27/2021, 7:28 PM

Alright. I'm not sure how to check a Fargate task allocated resources (I can't SSH into the container), and the AWS doesn't offer obvious source of this information. Do you maybe know how to check this?

Zanie

05/27/2021, 7:54 PM

I'll ping someone who's worked with ECS recently on our team.

Zanie

05/28/2021, 4:20 PM

Hey @Nikola Lusic -- we have confirmed that the CPU and memory are passed correctly to your task run. Since you define your own task definition we must pass things like this as overrides per run rather than on the task definition. You can use

aws ecs describe-tasks

to take a look at your task and see that the settings are correct https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html

👍 1

Zanie

05/28/2021, 4:20 PM

@Marvin archive "`ECSRun` memory and CPU settings are not set in the task definition"

Marvin

05/28/2021, 4:20 PM

https://github.com/PrefectHQ/prefect/issues/4599

6 Views

Open in Slack

Previous Next