We are having issues getting up and running with

FargateTaskEnvironment

. I think we did all according to the docs but when passing things through to AWS when it tries to run the flow we get the following error (on prefect cloud):

An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.

This is our environment:

flow.environment = FargateTaskEnvironment(
            executor=DaskExecutor(),
            labels=[f"reco-{self._config.env}"],
            launch_type="FARGATE",
            region="eu-west-1",
            taskRoleArn="arn:aws:iam::XXXXXXXX:role/prefect-task-role-manual",
            executionRoleArn="arn:aws:iam::XXXXXX:role/prefect-execution-role-manual",
            family="tst_flow",
            cluster="reco-dev-tasks-ecscluster-ECSCluster-13542I9PRAV48",
            networkConfiguration={
                "awsvpcConfiguration": {
                    "assignPublicIp": "ENABLED",
                    "subnets": [
                        "subnet-05fd20a54e43646xx",
                        "subnet-04e8a87c3f718dcxx",
                        "subnet-0545ba61d9f1b5exx",
                    ],
                    "securityGroups": ["sg-005d5df0d9d6594xx"],
                }
            },
            cpu="512",
            memory="3072",
            containerDefinitions={
                    "name": "flow-container",
                    "image": "image",
                    "command": [],
                    "environment": [],
                    "essential": True,
                    "memoryReservation": "3072",
                    "logConfiguration": {
                        "logDriver": "awslogs",
                        "options": {
                            "awslogs-group": "/ecs/prefect-runs-manual",
                            "awslogs-region": "eu-west-1",
                            "awslogs-stream-prefix": "prefect-flow-runs",
                            "awslogs-create-group": "true",
                        },
                    },
                }
        )

We also use Docker-based storage. When putting this config to the

FargateAgent

it can correctly run flows e.g. with a

LocalEnvironment

but then the whole config is in the agent and the same for all flows. The issue is somehow how the parameters are passed forward to AWS. Our code is very close to the example in the docs: https://docs.prefect.io/orchestration/execution/fargate_task_environment.html#examples But we tried it with

memory

and

memoryReservation

in container definitions but same error in both cases. Also tried to pass

containerDefinitions

as array. Any help would be greatly appreciated. Thanks 🙂

Hey @Michael Ludwig! What version of Prefect are you using?

Hey. It is

0.12.2

. The newest

Thanks!

@Michael Ludwig let me get a flow up an running with this environment. There’s a chance AWS changed their API again which would mean the examples need to be updated

What I also find curious is the example uses a dict and not an array. Because when looking in the code the first container definition [0] is referenced and used. Thanks for looking into it 🙂

Yeah that is definitely a typo 😅

My logs also speaking about a container called

flow

. When digging into the code it seems it is called

flow-container

. Maybe that is a hint?

[2020-07-09 14:55:55,903] ERROR - agent | Error while deploying flow: ClientException("An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.")

Interesting on the

flow

name. Still looking into it, having trouble getting mine running, the code is a bit old

I guess it is mainly about our confusion when it is right to use what. And if I understand all correctly we better specify it in the environment. Our plan is to have an fargate agent which can spin up a fargate task to run our flow

Yeah there is a lot of confusion around this environment schema and that’s why we are revamping it really soon! So you don’t actually need to use this environment in order to run your flows on fargate. A combination of the Fargate Agent and the default LocalEnvironment will run your flows on fargate for you as fargate tasks 🙂 This environment was added originally for some extra configuration options and it has since fallen out of use

@josh okay great the combination we had before and we also have no problem switching to it back. I just grabbed some logs when running the agent in verbose mode btw:

[2020-07-09 15:31:18,570] DEBUG - agent | Next query for flow runs in 0.5 seconds
[2020-07-09 15:31:18,581] DEBUG - agent | No task definition found
[2020-07-09 15:31:18,581] DEBUG - agent | Using image <http://XXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/tst-prefect-flows-2:latest|XXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/tst-prefect-flows-2:latest> for task definition
[2020-07-09 15:31:18,581] DEBUG - agent | Removing static Prefect envars from container_definitions_kwargs if exists
[2020-07-09 15:31:18,582] DEBUG - agent | Registering task definition prefect-task-99380786
[2020-07-09 15:31:18,805] ERROR - agent | Logging platform error for flow run 1f99ae9b-1764-470e-bd24-8a31a2fe6d8b
[2020-07-09 15:31:19,074] DEBUG - agent | Querying for flow runs
[2020-07-09 15:31:19,282] DEBUG - agent | No flow runs found
[2020-07-09 15:31:19,282] DEBUG - agent | Next query for flow runs in 1.0 seconds
[2020-07-09 15:31:19,332] ERROR - agent | Error while deploying flow: ClientException("An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.")
[2020-07-09 15:31:19,332] DEBUG - agent | Completed flow run submission (id: 1f99ae9b-1764-470e-bd24-8a31a2fe6d8b)

Yeah not sure where that

flow

name is coming from 🤔 I wonder if it’s using an old task definition. If you change your

family

to another string does it still show the same issue?

I tried to modify the code locally for the agent and add some prints. Very amateur debugging but at least I could not see that my stuff was printed. So I guess the correct

setup

method is not called:

def setup(self, flow: "Flow") -> None:  # type: ignore:
   ....
   print(self.task_definition_kwargs)

   boto3_c.register_task_definition(**self.task_definition_kwargs)