How can i use an ecs fargate cluster and run tasks in parall Prefect Community #prefect-server

Join Slack

How can i use an ecs fargate cluster and run tasks...

# prefect-server

Gagan Singh Saluja

01/30/2022, 8:17 PM

How can i use an ecs fargate cluster and run tasks in parallel? Like using fargate cluster for dask.

Anna Geller

01/30/2022, 8:35 PM

We don’t have any official tutorial about that yet but you can use the code example from here.

Copy code

executor=DaskExecutor(
        cluster_class=fargate_cluster,
        cluster_kwargs={
            "image": "annageller/prefect-dask-cloudprovider:latest",
            "n_workers": 2,
        },
        debug=True,
    ),

Anna Geller

01/30/2022, 8:35 PM

in short, you can create a Dask cluster on Fargate and parallelize execution across this cluster using mapping

Gagan Singh Saluja

01/31/2022, 5:58 AM

Thanks, i’ll try this

Gagan Singh Saluja

02/20/2022, 10:46 AM

Hi, you have mentioned in the link above that you have attached required dask permission in task execution role, where can i find list of those permissions to attach??

Anna Geller

02/20/2022, 11:32 AM

This Dask page provides the permissions in the docstrings https://cloudprovider.dask.org/en/latest/_modules/dask_cloudprovider/aws/ecs.html#FargateCluster

Copy code

{
  "Statement": [
    {
      "Action": [
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateSecurityGroup",
        "ec2:CreateTags",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:DeleteSecurityGroup",
        "ecs:CreateCluster",
        "ecs:DescribeTasks",
        "ecs:ListAccountSettings",
        "ecs:RegisterTaskDefinition",
        "ecs:RunTask",
        "ecs:StopTask",
        "ecs:ListClusters",
        "ecs:DescribeClusters",
        "ecs:DeleteCluster",
        "ecs:ListTaskDefinitions",
        "ecs:DescribeTaskDefinition",
        "ecs:DeregisterTaskDefinition",
        "iam:AttachRolePolicy",
        "iam:CreateRole",
        "iam:TagRole",
        "iam:PassRole",
        "iam:DeleteRole",
        "iam:ListRoles",
        "iam:ListRoleTags",
        "iam:ListAttachedRolePolicies",
        "iam:DetachRolePolicy",
        "logs:DescribeLogGroups",
        "logs:GetLogEvents",
        "logs:CreateLogGroup",
        "logs:PutRetentionPolicy"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

Gagan Singh Saluja

02/20/2022, 11:33 AM

Thanks a lot

Open in Slack

Previous Next