Hi Prefect devs, me again. I am oh so close to fi...
# ask-community
b
Hi Prefect devs, me again. I am oh so close to finish my deployment system for prefect for my company but am running into an issue. I have my agent deployed in a cluster and running successfully but then I am trying to execute a flow with a menial logging task to make sure everything works. It is running in the same cluster as the agent. When the task definition spins up the task, it is hit with an error:
Copy code
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 1 time(s): AccessDeniedException: User: arn:aws:sts::*****:assumed-rol...
Both my execution role and task role have admin access ( while i am debugging ) so i dont think it is a role based issue. Wondering if anyone has come across something like this before and can help out. Oh and this is running on ECS and fargate, I have attached my flow code in the ๐Ÿงต
Copy code
import prefect
from prefect.storage import S3, Docker
from prefect.run_configs import ECSRun
from prefect import task, Flow


TASK_ARN = "arn:aws:iam::***:role/ECSTaskS3ECRRole"
RUN_CONFIG = ECSRun(
    run_task_kwargs={
        "cluster": "prefect-agent-cluster",
        "networkConfiguration":{'awsvpcConfiguration': {'assignPublicIp': 'ENABLED', 'subnets': ['subnet-00637e8e1b0c3b2e8', 'subnet-02847af14d62b73c1'], 'securityGroups': []}}
    },
    labels=['s3-flow-storage'],
    task_role_arn=TASK_ARN,
    execution_role_arn='arn:aws:iam::***:role/ECSFullyLoaded',
    # image='prefecthq/prefect:latest-python3.8',
    # memory=512,
    # cpu=256
    )

STORAGE = Docker(
    registry_url='***.<http://dkr.ecr.ap-southeast-2.amazonaws.com/|dkr.ecr.ap-southeast-2.amazonaws.com/>', 
    image_name='prefect-tutorial', 
    image_tag='latest',
    dockerfile='./Dockerfile'
    )


@task
def say_hello():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>('This Worked!')



with Flow(
    "s3_pandas", 
    storage=STORAGE,
    run_config=RUN_CONFIG
    ) as flow:

    say_hello()

flow.register(project_name="aws-flow")
# flow.run()
z
Hi @Ben Muller - can you double check the ECR permissions configured for your image allow the IAM roles for your execution/task to pull images?https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policies.html
upvote 1
n
Another thing to check: if you're launching tasks in a private subnet you'll need to make sure it has a VPC routing table to route outbound traffic somewhere that can open a connection to ECR, where your flow is being stored (there might be other ways of doing this that don't include a public subnet but I'm not sure)
upvote 1
b
Thanks @Zach Angell - trying this now. @nicholas - this is a bit out of my wheelhouse, do you have any documentation that you can point me to that can explain this a bit ?
Hey @Zach Angell - that didnt work unfortunately ....
n
I don't have any prefect-specific documentation (I don't think it's needed anyway) but I have this blog post from the ECS team and this documentation on ECS network configuration which might be helpful
b
Thanks so much, I will give it a read. wish me luck!
n
Good luck! ๐Ÿ˜„
z
Perhaps https://aws.amazon.com/premiumsupport/knowledge-center/ecs-tasks-pull-images-ecr-repository/ and https://aws.amazon.com/blogs/compute/setting-up-aws-privatelink-for-amazon-ecs-and-amazon-ecr/ may be helpful as well. I'd dig into the 'AccessDeniedException' and see if you can determine which case causes that specific error.
upvote 2
b
cool cheers, a lot more involved than I would have thought. ๐Ÿ˜ž
Thanks for all the help, decided to change the storage to S3 for ease!
๐Ÿ‘ 2