https://prefect.io logo
#prefect-community
Title
# prefect-community
v

Vamsi Reddy

05/24/2022, 2:33 PM
Hi there, we are using an ECS agent to run our flows. is it possible to set the cpu and memory per flow. I tried setting the cpu and memory for the flow in ECSRun config but the task that gets spun up still uses the default that was specified in the yaml file i.e cpu 1024 and memory 2048. i want to change the cpu and memory per flow. my current ecs run config is as follows:
Copy code
RUN_CONFIG = ECSRun(
    labels=["dev"],
    task_role_arn="arn:aws:iam::xxxxx:role/prefectTaskRole",
    execution_role_arn="arn:aws:iam::xxxxx:role/prefectECSAgentTaskExecutionRole",
    image="<http://xxxxxx.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:latest|xxxxxx.dkr.ecr.us-east-1.amazonaws.com/prefect-orchestration:latest>",      # this is our custom image that also has our flows
    run_task_kwargs=dict(cluster="prefectEcsClusterDev", launchType="FARGATE", overrides=dict(
        containerOverrides=[
            dict(
                name="flow",
                cpu=4096,
                memory=8192,
            )])),
    cpu=4096,                    #tried specifying here but still no luck
    memory=8192
)
k

Kevin Kho

05/24/2022, 2:55 PM
Inside the
ECSRun(…)
will already have the highest priority and take effect. Did you re-register?
v

Vamsi Reddy

05/24/2022, 2:57 PM
yes i tried re-registering the flow
do i need to restart the ecs agent ?
k

Kevin Kho

05/24/2022, 2:58 PM
This looks pretty right to me already. Will try it in a bit.
👍 1
No you should not need to restart the agent
What is your Prefect version btw?
v

Vamsi Reddy

05/24/2022, 3:03 PM
0.15.13
shall i upgrade my prefect version?
our ecs agent shows core version as 1.0.0
k

Kevin Kho

05/24/2022, 3:26 PM
I didn’t see anything in the changelogs that would affect this behavior
👍 1
v

Vamsi Reddy

05/24/2022, 4:08 PM
@Kevin Kho any luck with getting this to work? just fyi we followed this tutorial to setup our ecs agent : https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df we also now have our own custom image that we deployed with our flows and other packages in it.
k

Kevin Kho

05/24/2022, 4:09 PM
nearly there
👍 1
How do you know what the cpu and memory are for the ECS task? Trying to poke around in the UI but can’t find
v

Vamsi Reddy

05/24/2022, 4:22 PM
you can click on task definition….then go to the json tab
k

Kevin Kho

05/24/2022, 4:22 PM
So I see the default values in the task definition. I am wondering if this is overridden during run time. I left a message for the team and then will update you
v

Vamsi Reddy

05/24/2022, 4:24 PM
thank you very much for digging deeper into this 🙏
k

Kevin Kho

05/24/2022, 4:25 PM
This looks broken and there is an open issue. You’d need to use a task definition yaml for now
v

Vamsi Reddy

05/24/2022, 4:29 PM
i do see a pr merged on that one is there a chance that this is fixed in the newer release?
k

Kevin Kho

05/24/2022, 4:30 PM
No that PR merged is a separate thing to allow for SPOT instances
👍 1
You might be able to fix it though if you want to take a stab
👍 1
6 Views