Hey, i’m new to using prefect - but i’ve been trying to follow along with this tutorial: https://towardsdatascience.com/distributed-data-pipelines-made-easy-with-aws-eks-and-prefect-106984923b30 I have most of it working, pods get spun up when i run a flow. However, they are unable to pull the image from ECR that I have pushed. I keep getting the error (i have replaced the account number and repo name)

Message: Failed to pull image "<http://z.dkr.ecr.us-east-1.amazonaws.com/r:latest|z.dkr.ecr.us-east-1.amazonaws.com/r:latest>": rpc error: code = NotFound desc = failed to pull and unpack image "<http://z.dkr.ecr.us-east-1.amazonaws.com/r:latest|z.dkr.ecr.us-east-1.amazonaws.com/r:latest>": failed to resolve reference "<http://z.dkr.ecr.us-east-1.amazonaws.com/r:latest|z.dkr.ecr.us-east-1.amazonaws.com/r:latest>": <http://z.dkr.ecr.us-east-1.amazonaws.com/r:latest|z.dkr.ecr.us-east-1.amazonaws.com/r:latest>: not found

I have followed the tutorial pretty closely, although some stuff what slightly outdated so I used the newer paradigms that were introduced. Any insight on how I could configure my EKS to be able to pull images from the ECR would be appreciated. I also did all of this on my master admin account, so I would think that permissions shouldn’t be an issue

Interesting! So it looks like the IAM role used by your Fargate profile doesn’t have access to pull images from ECR. Usually this permission is added automatically. When you use this command:

eksctl get fargateprofile --cluster your_cluster_name -o yaml

You should get info about which podExecutionRoleARN is used and you can check in your IAM console what permissions are attached, and if ECR is missing, perhaps you can add it? Still weird, ECR permission is granted by default, do you happen to use a different region in your ECR image as opposed to the region of the cluster?

They should both use us-east-1

did you check the IAM role?

Yeah so the IAM role is FargatePodExecuctionRole, I went to it in IAM last night and attached a JSON policy of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetAuthorizationToken"
      ],
      "Resource": "*"
    }
  ]
}

Based on this, it looks like the permissions need to be a bit broader:

"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:GetLifecyclePolicy",
"ecr:GetLifecyclePolicyPreview",
"ecr:ListTagsForResource",
"ecr:DescribeImageScanFindings"

how did you create your cluster? Was your fargate profile created automatically or manually?

it was created automatically

as a workaround, you could use Kubernetes secret as shown here, but this secret token is short lived so IAM role is recommended

let me update the IAM permissions and see if this works, thanks!

If nothing else works, you can try everything from scratch (new cluster, new ECR repo), ideally in a different region e.g. us-east-2 You can also contact AWS if you have a paid support plan Lastly, you can check the cloud formation template - eksctl deploys all resources as CloudFormation stack under the hood - you can check the outputs there to check if everything worked correctly

And.. this ended up being a typo on my end 😓. Thanks for the help getting this sorted!