https://prefect.io logo
Channels
Powered by
a

ale

03/11/2021, 6:57 PM
Hey folks 🙂 I've started experimenting with ECS agent to replace our current Fargate agent, but I'm having issues with AWS permissions. Is there a place where I can find the minimum set of permissions that an ECS agent needs to successfully deploy a flow as an ECS task?
z

Zanie

03/11/2021, 7:00 PM
Hmm I know this was recently added in https://github.com/PrefectHQ/prefect/pull/4111/files
Not sure where this is hiding in the live docs -.-
a

ale

03/11/2021, 7:05 PM
Thanks @Zanie simple smile
z

Zanie

03/11/2021, 7:06 PM
It sounds like this isn't actually documented after talking to Jim, he thinks something like
ECS register, deregister, and run tasks, and some EC2 calls by default to infer things about the vpc/subnets (describe vpcs, describe subnets)
If you're running an older version there may be more since we recently reduced the required permissions
a

ale

03/11/2021, 7:07 PM
We’re still on 0.13.13 with Fargate Agent but we definitely want to upgrade to a more recent version and use ECS Agent
But as I said, it’s a bit difficult to get a working agent due to lack of docs regarding required permissions. Lots of trials and errors 🙂
z

Zanie

03/11/2021, 7:15 PM
We'd happily accept a PR improving it 😄 it's on the to-do list for sure.
@Marvin open "Document 
ECSAgent
required IAM permissions"
m

Marvin

03/11/2021, 7:16 PM
a

ale

03/11/2021, 8:30 PM
Once I get a full understanding of ECS agent permissions for AWS I'm going to write some docs and submit a PR
r

Robert Bastian

03/12/2021, 12:16 AM
Here are the permissions that we use for ESC/Fargate. Our agent (0.14.6) has the following permissions to use boto3 for submitting tasks - AmazonS3FullAccess and AmazonECSFull Access. Our task_run_arn has the following permissions: AmazonS3FullAccess and AmazonEC2ContainerRegistryFullAccess. Our execution_run_arn has AmazonEcsTaskExecutionRole. It could be that we’re too lenient on permissions and could scale them back - its on our todo list but this got us unblocked. HTH.
c

ciaran

03/25/2021, 2:22 PM
Hey @ale did you get a chance to figure out the full 'minimum' permissions needed?
a

ale

03/25/2021, 2:25 PM
Hey @ciaran simple smile Not yet, unfortunately
c

ciaran

03/25/2021, 2:26 PM
Oh 😭 No worries, are you just going with the full ecs access at the mo?
a

ale

03/25/2021, 3:02 PM
Nope, we just postponed the Prefect Server upgrade and sticked with 
0.13.13
😅
r

Roger L.

04/07/2021, 2:09 PM
Hello @ciaran @ale. I have given 
ecs:*
permissions to my ECSAgent and I have checked CloudTrail logs to find out which actions is the agent performing. This is my current policy:
Copy code
{
  "Statement": [
    {
      "Action": [
        "ecs:RunTask",
        "ecs:RegisterTaskDefinition",
        "ecs:DeregisterTaskDefinition"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": "iam:PassRole",
      "Effect": "Allow",
      "Resource": [
        "arn:aws:iam::4XXXXX:role/PrefectFlowStagingTaskRole",
        "arn:aws:iam::4XXXXX:role/PrefectFlowStagingTaskExecutionRole"
      ]
    }
  ],
  "Version": "2012-10-17"
}
Where 
PrefectFlowStagingTaskRole
and 
PrefectFlowStagingTaskExecutionRole
are the roles that the agent passes to the deployed task on Fargate.
a

ale

04/07/2021, 2:16 PM
Thanks a lot @Roger L.!!!
r

Roger L.

04/07/2021, 2:17 PM
Btw I am checking this comment from potter420: https://github.com/PrefectHQ/prefect/pull/4302
c

ciaran

04/07/2021, 2:18 PM
@Roger L. Nice one! 🦜 Cloud trail is a great shout
r

Roger L.

04/07/2021, 2:18 PM
I might be missing something like 
"ecs:StopTask"
, but hope it helps guys. I am still on it 😛
c

ciaran

04/07/2021, 2:19 PM
Certainly a great starting point thanks!
r

Roger L.

04/07/2021, 2:19 PM
You are welcome 🙂
c

ciaran

04/07/2021, 2:30 PM
@Roger L. out of interest, how're you deploying this all? I'd love to pick folk's brains about automated Prefect cluster deployment on AWS, it's something we're working on at the mo
a

ale

04/07/2021, 2:51 PM
We’re using CloudFormation for provisioning all the required resources. Prefect Server + UI + Agents are all part of an ECS service
z

Zanie

04/07/2021, 2:52 PM
c

ciaran

04/07/2021, 2:54 PM
Cool, we're using CDK over on the pangeo-forge project, so I guess just CFN in the end too
r

Roger L.

04/07/2021, 3:16 PM
We are using Prefect Cloud, our agents are dockerized and run on fargate. The IaC is managed with Terraform.
✅ 1
Powered by