https://prefect.io logo
Title
m

Michael Ludwig

07/09/2020, 2:32 PM
We are having issues getting up and running with
FargateTaskEnvironment
. I think we did all according to the docs but when passing things through to AWS when it tries to run the flow we get the following error (on prefect cloud):
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.
This is our environment:
flow.environment = FargateTaskEnvironment(
            executor=DaskExecutor(),
            labels=[f"reco-{self._config.env}"],
            launch_type="FARGATE",
            region="eu-west-1",
            taskRoleArn="arn:aws:iam::XXXXXXXX:role/prefect-task-role-manual",
            executionRoleArn="arn:aws:iam::XXXXXX:role/prefect-execution-role-manual",
            family="tst_flow",
            cluster="reco-dev-tasks-ecscluster-ECSCluster-13542I9PRAV48",
            networkConfiguration={
                "awsvpcConfiguration": {
                    "assignPublicIp": "ENABLED",
                    "subnets": [
                        "subnet-05fd20a54e43646xx",
                        "subnet-04e8a87c3f718dcxx",
                        "subnet-0545ba61d9f1b5exx",
                    ],
                    "securityGroups": ["sg-005d5df0d9d6594xx"],
                }
            },
            cpu="512",
            memory="3072",
            containerDefinitions={
                    "name": "flow-container",
                    "image": "image",
                    "command": [],
                    "environment": [],
                    "essential": True,
                    "memoryReservation": "3072",
                    "logConfiguration": {
                        "logDriver": "awslogs",
                        "options": {
                            "awslogs-group": "/ecs/prefect-runs-manual",
                            "awslogs-region": "eu-west-1",
                            "awslogs-stream-prefix": "prefect-flow-runs",
                            "awslogs-create-group": "true",
                        },
                    },
                }
        )
We also use Docker-based storage. When putting this config to the
FargateAgent
it can correctly run flows e.g. with a
LocalEnvironment
but then the whole config is in the agent and the same for all flows. The issue is somehow how the parameters are passed forward to AWS. Our code is very close to the example in the docs: https://docs.prefect.io/orchestration/execution/fargate_task_environment.html#examples But we tried it with
memory
and
memoryReservation
in container definitions but same error in both cases. Also tried to pass
containerDefinitions
as array. Any help would be greatly appreciated. Thanks 🙂
👀 1
d

Dylan

07/09/2020, 2:48 PM
Hey @Michael Ludwig! What version of Prefect are you using?
m

Michael Ludwig

07/09/2020, 2:48 PM
Hey. It is
0.12.2
. The newest
d

Dylan

07/09/2020, 2:50 PM
Thanks!
I am not an AWS guy but I will do my best to help you out. I’m looking through your config / our code now
j

josh

07/09/2020, 2:52 PM
@Michael Ludwig let me get a flow up an running with this environment. There’s a chance AWS changed their API again which would mean the examples need to be updated
m

Michael Ludwig

07/09/2020, 2:53 PM
What I also find curious is the example uses a dict and not an array. Because when looking in the code the first container definition [0] is referenced and used. Thanks for looking into it 🙂
j

josh

07/09/2020, 2:53 PM
Yeah that is definitely a typo 😅
m

Michael Ludwig

07/09/2020, 2:57 PM
My logs also speaking about a container called
flow
. When digging into the code it seems it is called
flow-container
. Maybe that is a hint?
[2020-07-09 14:55:55,903] ERROR - agent | Error while deploying flow: ClientException("An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.")
The agent picking up the flow is just started with this btw:
REGION_NAME=eu-west-1 prefect agent start fargate -t XXXXX --label "reco-dev"
j

josh

07/09/2020, 3:17 PM
Interesting on the
flow
name. Still looking into it, having trouble getting mine running, the code is a bit old
Quick question, @Michael Ludwig is there a reason why you would need to use the FargateTaskEnvironment over the configuration you can provide to the Fargate Agent for the initial prefect-task? I’m asking because this environment is being deprecated soon in favor of a better pattern
m

Michael Ludwig

07/09/2020, 3:26 PM
I guess it is mainly about our confusion when it is right to use what. And if I understand all correctly we better specify it in the environment. Our plan is to have an fargate agent which can spin up a fargate task to run our flow
It did work before when specifying the options at the
FargateAgent
though. So I suspect maybe a bug in
FargateTaskEnvironment
or a small difference in how you use it compared to
FargateAgent
j

josh

07/09/2020, 3:29 PM
Yeah there is a lot of confusion around this environment schema and that’s why we are revamping it really soon! So you don’t actually need to use this environment in order to run your flows on fargate. A combination of the Fargate Agent and the default LocalEnvironment will run your flows on fargate for you as fargate tasks 🙂 This environment was added originally for some extra configuration options and it has since fallen out of use
It appears as if the Fargate Agent being more maintained has allowed it to keep up with changes in the boto3 library where this environment has fallen behind
m

Michael Ludwig

07/09/2020, 3:33 PM
@josh okay great the combination we had before and we also have no problem switching to it back. I just grabbed some logs when running the agent in verbose mode btw:
[2020-07-09 15:31:18,570] DEBUG - agent | Next query for flow runs in 0.5 seconds
[2020-07-09 15:31:18,581] DEBUG - agent | No task definition found
[2020-07-09 15:31:18,581] DEBUG - agent | Using image <http://XXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/tst-prefect-flows-2:latest|XXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/tst-prefect-flows-2:latest> for task definition
[2020-07-09 15:31:18,581] DEBUG - agent | Removing static Prefect envars from container_definitions_kwargs if exists
[2020-07-09 15:31:18,582] DEBUG - agent | Registering task definition prefect-task-99380786
[2020-07-09 15:31:18,805] ERROR - agent | Logging platform error for flow run 1f99ae9b-1764-470e-bd24-8a31a2fe6d8b
[2020-07-09 15:31:19,074] DEBUG - agent | Querying for flow runs
[2020-07-09 15:31:19,282] DEBUG - agent | No flow runs found
[2020-07-09 15:31:19,282] DEBUG - agent | Next query for flow runs in 1.0 seconds
[2020-07-09 15:31:19,332] ERROR - agent | Error while deploying flow: ClientException("An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.")
[2020-07-09 15:31:19,332] DEBUG - agent | Completed flow run submission (id: 1f99ae9b-1764-470e-bd24-8a31a2fe6d8b)
j

josh

07/09/2020, 3:38 PM
Yeah not sure where that
flow
name is coming from 🤔 I wonder if it’s using an old task definition. If you change your
family
to another string does it still show the same issue?
m

Michael Ludwig

07/09/2020, 3:39 PM
I tried to modify the code locally for the agent and add some prints. Very amateur debugging but at least I could not see that my stuff was printed. So I guess the correct
setup
method is not called:
def setup(self, flow: "Flow") -> None:  # type: ignore:
   ....
   print(self.task_definition_kwargs)

   boto3_c.register_task_definition(**self.task_definition_kwargs)
So maybe the FargateAgent somehow does something before the
setup
stuff si correctly done
And yeah I changed already family, it is the same
Another hint into that direction is that FargateAgent tries to create a container called
flow
instead of
flow-container
prefect/agent/fargate/agent.py
container_definitions = [
            {
                "name": "flow",
                "image": image,
...