https://prefect.io logo
Title
m

Michael Ludwig

07/20/2020, 11:44 AM
We are using a FargateAgent and currently experimenting with different memory and CPU settings for the flows it fires off. We have the issue that we can’t change the resources anymore because it fails to recognize that it needs to recreate and ECS TaskDefinition when the memory was changed. So even with setting the right values it starts the old task definition with old memory settings:
class ECSFargateAgent:
    def __init__(self, config: PrefectAgentConfig):
        self._env = config.env
        self._agent = FargateAgent(
            labels=[f"reco-{self._env}"],
            enable_task_revisions=True,
            launch_type="FARGATE",
            taskRoleArn=config.task_role_arn,
            executionRoleArn=config.execution_role_arn,
            cluster=config.ecs_cluster_arn,
            networkConfiguration={
                "awsvpcConfiguration": {
                    "assignPublicIp": "ENABLED",
                    "subnets": config.subnets,
                    "securityGroups": [config.security_group],
                }
            },
            cpu="1024",  # 1 vCPU
            memory="3072",  # 3 GB
            containerDefinitions=[
                {
                    "logConfiguration": {
                        "logDriver": "awslogs",
                        "options": {
                            "awslogs-group": config.log_group,
                            "awslogs-region": "eu-west-1",
                            "awslogs-stream-prefix": "flows",
                        },
                    },
                }
            ],
        )

    def run(self):
        """Start the agent"""
        self._agent.start()
e.g. we switched
memory
from 16GB to 3GB but the agent still fires off flows with 16GB. Only deleting the old task definitions manually solves this for us. Anybody seen something similiar or has a solution?
b

Ben Davison

07/20/2020, 12:10 PM
I think
enable_task_revisions
is doing the opposite of what you expect it todo: https://github.com/PrefectHQ/prefect/blob/653478e6140a79a54fb4cba5d5f1a1f92e236ae5/src/prefect/agent/fargate/agent.py#L83-L89 My reading of that, is that if you set it to
True
it will use the flow_name instead of the flow_id which won't change.
m

Michael Ludwig

07/20/2020, 12:30 PM
Good hint but I think I had that before and it spammed our task definitions with an ever-growing list. So a new revision would be fine for me but the problem is that it just does not create a new revision. It does not realize that there is a change. So I am not really sure if I want to switch that option off because it does not do what I would expect it to do. But I guess because the flow name and version didn’t change it does not figure out that it needs to recreate a new revision because it probably does not compare these. It uses the Flow ID in the tags though. Probably not enough to figure out a diff. It could also be that an IAM role is switched out and then it uses the old one.