Michael Ludwig
07/09/2020, 2:32 PMFargateTaskEnvironment
. I think we did all according to the docs but when passing things through to AWS when it tries to run the flow we get the following error (on prefect cloud):
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.
This is our environment:
flow.environment = FargateTaskEnvironment(
executor=DaskExecutor(),
labels=[f"reco-{self._config.env}"],
launch_type="FARGATE",
region="eu-west-1",
taskRoleArn="arn:aws:iam::XXXXXXXX:role/prefect-task-role-manual",
executionRoleArn="arn:aws:iam::XXXXXX:role/prefect-execution-role-manual",
family="tst_flow",
cluster="reco-dev-tasks-ecscluster-ECSCluster-13542I9PRAV48",
networkConfiguration={
"awsvpcConfiguration": {
"assignPublicIp": "ENABLED",
"subnets": [
"subnet-05fd20a54e43646xx",
"subnet-04e8a87c3f718dcxx",
"subnet-0545ba61d9f1b5exx",
],
"securityGroups": ["sg-005d5df0d9d6594xx"],
}
},
cpu="512",
memory="3072",
containerDefinitions={
"name": "flow-container",
"image": "image",
"command": [],
"environment": [],
"essential": True,
"memoryReservation": "3072",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/prefect-runs-manual",
"awslogs-region": "eu-west-1",
"awslogs-stream-prefix": "prefect-flow-runs",
"awslogs-create-group": "true",
},
},
}
)
We also use Docker-based storage. When putting this config to the FargateAgent
it can correctly run flows e.g. with a LocalEnvironment
but then the whole config is in the agent and the same for all flows.
The issue is somehow how the parameters are passed forward to AWS. Our code is very close to the example in the docs: https://docs.prefect.io/orchestration/execution/fargate_task_environment.html#examples
But we tried it with memory
and memoryReservation
in container definitions but same error in both cases. Also tried to pass containerDefinitions
as array.
Any help would be greatly appreciated. Thanks 🙂Dylan
07/09/2020, 2:48 PMMichael Ludwig
07/09/2020, 2:48 PM0.12.2
. The newestDylan
07/09/2020, 2:50 PMjosh
07/09/2020, 2:52 PMMichael Ludwig
07/09/2020, 2:53 PMjosh
07/09/2020, 2:53 PMMichael Ludwig
07/09/2020, 2:57 PMflow
. When digging into the code it seems it is called flow-container
. Maybe that is a hint?
[2020-07-09 14:55:55,903] ERROR - agent | Error while deploying flow: ClientException("An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.")
REGION_NAME=eu-west-1 prefect agent start fargate -t XXXXX --label "reco-dev"
josh
07/09/2020, 3:17 PMflow
name. Still looking into it, having trouble getting mine running, the code is a bit oldMichael Ludwig
07/09/2020, 3:26 PMFargateAgent
though. So I suspect maybe a bug in FargateTaskEnvironment
or a small difference in how you use it compared to FargateAgent
josh
07/09/2020, 3:29 PMMichael Ludwig
07/09/2020, 3:33 PM[2020-07-09 15:31:18,570] DEBUG - agent | Next query for flow runs in 0.5 seconds
[2020-07-09 15:31:18,581] DEBUG - agent | No task definition found
[2020-07-09 15:31:18,581] DEBUG - agent | Using image <http://XXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/tst-prefect-flows-2:latest|XXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/tst-prefect-flows-2:latest> for task definition
[2020-07-09 15:31:18,581] DEBUG - agent | Removing static Prefect envars from container_definitions_kwargs if exists
[2020-07-09 15:31:18,582] DEBUG - agent | Registering task definition prefect-task-99380786
[2020-07-09 15:31:18,805] ERROR - agent | Logging platform error for flow run 1f99ae9b-1764-470e-bd24-8a31a2fe6d8b
[2020-07-09 15:31:19,074] DEBUG - agent | Querying for flow runs
[2020-07-09 15:31:19,282] DEBUG - agent | No flow runs found
[2020-07-09 15:31:19,282] DEBUG - agent | Next query for flow runs in 1.0 seconds
[2020-07-09 15:31:19,332] ERROR - agent | Error while deploying flow: ClientException("An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified.")
[2020-07-09 15:31:19,332] DEBUG - agent | Completed flow run submission (id: 1f99ae9b-1764-470e-bd24-8a31a2fe6d8b)
josh
07/09/2020, 3:38 PMflow
name is coming from 🤔 I wonder if it’s using an old task definition. If you change your family
to another string does it still show the same issue?Michael Ludwig
07/09/2020, 3:39 PMsetup
method is not called:
def setup(self, flow: "Flow") -> None: # type: ignore:
....
print(self.task_definition_kwargs)
boto3_c.register_task_definition(**self.task_definition_kwargs)
setup
stuff si correctly doneflow
instead of flow-container
prefect/agent/fargate/agent.py
container_definitions = [
{
"name": "flow",
"image": image,
...