How can I give flows that run on my AWS ECS Fargate a cluste Prefect Community #prefect-aws

How can I give flows that run on my AWS ECS Fargat...

James Gatter

04/04/2023, 3:59 PM

How can I give flows that run on my AWS ECS Fargate a cluster EFS/EBS volume to temporarily write files to? I have tried using the

task_customizations

field of

ECSTask()

, but it seems like the fields I want to modify (`volumes`/`mountPoints`) are unsupported (see thread).

👀 1

James Gatter

04/04/2023, 4:03 PM

Trying to add patches

Copy code

[
        {
            "op": "add",
            "path": "/overrides/containerOverrides/0/mountPoints",
            "value": [{
                "containerPath": "/mnt/data",
                "readOnly": False,
                "sourceVolume": "my-ebs-volume"
            }]
        },
        {
            "op": "add",
            "path": "/volumes",
            "value": [{
                "name": "my-ebs-volume",
                "awsElasticBlockStore": {
                    "volumeId": None,
                    "volumeType": "gp2",
                    "size": 100,
                    "iops": None,
                    "encrypted": None,
                    "deleteOnTermination": True
                }
            }]
        }
    ]

to modify:

Copy code

{
  "overrides": {
    "containerOverrides": [
      {
        "name": "prefect",
        "environment": [
          {
            "name": "PREFECT_API_URL",
            "value": "<https://api.prefect.cloud/api/accounts/xxxx/workspaces/xxxx>"
          },
          {
            "name": "PREFECT_API_KEY",
            "value": "xxxxxxxxxxxxxxx"
          },
          {
            "name": "PREFECT_LOGGING_LEVEL",
            "value": "INFO"
          },
          {
            "name": "PREFECT__FLOW_RUN_ID",
            "value": "2d6da497fdf243d7902df035b27fa113"
          }
        ],
        "command": [
          "python",
          "-m",
          "prefect.engine"
        ],
        "memory": 65536,
        "cpu": 16384
      }
    ],
    "executionRoleArn": "arn:aws:iam::xxxxxxxxxxxx:role/cyflows_ecs_execution_role",
    "taskRoleArn": "arn:aws:iam::xxxxxxxxxxxx:role/cyflows_ecs_task_role",
    "memory": "65536",
    "cpu": "16384"
  },
  "tags": [
    {
      "key": "<http://prefect.io/flow-run-id|prefect.io/flow-run-id>",
      "value": "2d6da497-fdf2-43d7-902d-f035b27fa113"
    },
    {
      "key": "<http://prefect.io/flow-run-name|prefect.io/flow-run-name>",
      "value": "lambda155-loracus"
    },
    {
      "key": "<http://prefect.io/version|prefect.io/version>",
      "value": "2.9.0"
    },
    {
      "key": "<http://prefect.io/deployment-name|prefect.io/deployment-name>",
      "value": "dev-standalone"
    },
    {
      "key": "<http://prefect.io/deployment-updated|prefect.io/deployment-updated>",
      "value": "2023-04-03T20:16:44.723962Z"
    },
    {
      "key": "<http://prefect.io/flow-name|prefect.io/flow-name>",
      "value": "ild6_analysis"
    }
  ],
  "taskDefinition": "arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task-definition/prefect__ild6_analysis__dev-standalone:8",
  "cluster": "prefect-fargate",
  "launchType": "FARGATE",
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": [
        "subnet-xxxxxxxxxxx"
      ],
      "assignPublicIp": "ENABLED"
    }
  }
}

but I receive:

Copy code

Submission failed. botocore.exceptions.ParamValidationError: Parameter validation failed: Unknown parameter in input: "volumes", must be one of: capacityProviderStrategy, cluster, count, enableECSManagedTags, enableExecuteCommand, group, launchType, networkConfiguration, overrides, placementConstraints, placementStrategy, platformVersion, propagateTags, referenceId, startedBy, tags, taskDefinition Unknown parameter in overrides.containerOverrides[0]: "mountPoints", must be one of: name, command, environment, environmentFiles, cpu, memory, memoryReservation, resourceRequirements

James Gatter

04/04/2023, 4:04 PM

For this workflow I need to be able to write large files (a few ~4GB) to disk

James Gatter

04/04/2023, 4:17 PM

I'm guessing Prefect is wrapping this boto3 function: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/start_task.html

James Gatter

04/04/2023, 4:22 PM

From what I gather, one thing I could do is modify the task definition referenced by

taskDefinition

, but doing this manually isn't ideal for a deployment pipeline with many different workflows.

James Gatter

04/04/2023, 4:34 PM

I used a temporary EBS volume in this attempt, but I'm guessing it's better to have all deployment task definitions reference a single, shared EFS volume so they can share intermediate outputs.

Emil Christensen

04/04/2023, 6:01 PM

Hi @James Gatter 👋 My understanding is that

task_customizations

doesn’t allow you to override the task definition itself, only arguments to the actual run command. It does seem like the only supported way to specify volumes and mount points is to produce a custom task definition.

Emil Christensen

04/04/2023, 6:02 PM

If you wouldn’t mind opening up a feature request, we’d be happy to take a look.

James Gatter

04/04/2023, 6:37 PM

Opened https://github.com/PrefectHQ/prefect-aws/issues/232

🙏 1

3 Views

Open in Slack

Previous Next