<@ULVA73B9P> What permissions does the task run ro...
# ask-marvin
j
@Marvin What permissions does the task run role need in a ECS push pool with a temporary cluster dask task runner need (to create & manage the custer) dask_cloudprovider.aws.FargateCluster
m
thought for 70.3 seconds
Great question. Short answer: the IAM role assumed by your Prefect ECS task (the “task run role”) must be able to create and manage the Dask scheduler/worker tasks that dask_cloudprovider launches on your behalf. That means ECS, Logs, IAM PassRole, and (optionally) EC2 permissions if you let Dask create networking resources for you. What you need depends on whether you provide existing infra (ECS cluster, subnets, security group, roles) to FargateCluster or let it create them. Minimum permissions typically required by the task run role - ECS (create/manage cluster, task definitions, and tasks) - ecs:CreateCluster, ecs:DeleteCluster, ecs:DescribeClusters, ecs:ListClusters - ecs:RegisterTaskDefinition, ecs:DeregisterTaskDefinition, ecs:DescribeTaskDefinition - ecs:RunTask, ecs:StopTask, ecs:DescribeTasks, ecs:ListTasks - If FargateCluster uses a service for the scheduler: ecs:CreateService, ecs:UpdateService, ecs:DeleteService, ecs:DescribeServices - IAM (to pass the task/execution roles used by the Dask scheduler/worker tasks) - iam:PassRole on the specific ARNs you’ll pass to FargateCluster (task_role_arn, execution_role_arn) - CloudWatch Logs (for scheduler/worker logs) - logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents, logs:DescribeLogGroups, logs:DescribeLogStreams - EC2 (only if you allow FargateCluster to create or tweak security groups; otherwise just Describe*) - ec2:DescribeVpcs, ec2:DescribeSubnets, ec2:DescribeSecurityGroups - ec2:CreateSecurityGroup, ec2:DeleteSecurityGroup - ec2:AuthorizeSecurityGroupIngress, ec2:RevokeSecurityGroupIngress, ec2:AuthorizeSecurityGroupEgress, ec2:RevokeSecurityGroupEgress Example IAM policy for the task run role - Scope these down to your specific resources where possible; “*” shown for brevity.
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ECSCore",
      "Effect": "Allow",
      "Action": [
        "ecs:CreateCluster",
        "ecs:DeleteCluster",
        "ecs:DescribeClusters",
        "ecs:ListClusters",
        "ecs:RegisterTaskDefinition",
        "ecs:DeregisterTaskDefinition",
        "ecs:DescribeTaskDefinition",
        "ecs:RunTask",
        "ecs:StopTask",
        "ecs:DescribeTasks",
        "ecs:ListTasks",
        "ecs:CreateService",
        "ecs:UpdateService",
        "ecs:DeleteService",
        "ecs:DescribeServices"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Logs",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": "*"
    },
    {
      "Sid": "EC2NetworkingIfManagingSG",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:CreateSecurityGroup",
        "ec2:DeleteSecurityGroup",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupEgress"
      ],
      "Resource": "*"
    },
    {
      "Sid": "PassExecutionAndTaskRoles",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": [
        "arn:aws:iam::<ACCOUNT_ID>:role/<YourDaskExecutionRole>",
        "arn:aws:iam::<ACCOUNT_ID>:role/<YourDaskTaskRole>"
      ]
    }
  ]
}
Don’t forget the ECS task execution role - This is separate from the “task run role.” The execution role used by the Dask scheduler/worker tasks must be able to pull images and write logs, e.g.: - ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer (if using private ECR) - logs:CreateLogStream, logs:PutLogEvents - AWS creates/uses the service-linked role AWSServiceRoleForECS for ENIs; you typically do not need ec2:CreateNetworkInterface permissions on your task role. Tips to minimize permissions
- Provide FargateCluster with existing resources and you can drop the “create/delete” permissions: - ECS: use an existing ECS cluster name/ARN - Networking: pass existing subnets and a security group ID - Roles: pass pre-created task_role_arn and execution_role_arn If helpful, here’s how you’d wire those into FargateCluster:
Copy code
from dask_cloudprovider.aws import FargateCluster
from dask.distributed import Client

cluster = FargateCluster(
    region_name="us-east-1",
    vpc_id="vpc-xxxxxxxx",
    subnets=["subnet-aaaa", "subnet-bbbb"],
    security_groups=["sg-cccccccc"],
    task_role_arn="arn:aws:iam::<acct>:role/YourDaskTaskRole",
    execution_role_arn="arn:aws:iam::<acct>:role/YourDaskExecutionRole",
    image="public.ecr.aws/dask/dask:latest"
)
client = Client(cluster)
Docs and references - Prefect ECS work pools: Prefect ECS compute docs - dask-cloudprovider Fargate: FargateCluster docs - dask-cloudprovider repo (issues/examples): dask-cloudprovider on GitHub Questions for you - Do you want FargateCluster to create the ECS cluster and security group, or will you supply existing ones? - Are your Dask images in private ECR? If so, confirm your execution role has the ECR pull permissions above. If you share which parts you plan to manage vs. auto-create, I can trim this policy down to a tighter, least-privilege version for your setup.