Hey all Having issues getting the DockerAgent attempting to

Question

Hey all Having issues getting the DockerAgent attempting to do some DockerRun flows not being able to access the registry We have a set of images in an Amazon ECR location but I cant see any registry setting options in the DockerRun or DockerAgent and having the `AWS ACCESS KEY ID` and `AWS SECRET ACCESS KEY` set doesnt seem to be what prefect is looking for and it doesnt use them to authenticate Does prefect only support public images

Jan Marais · Answer

Is your Docker client authenticated with ECR

Jan Marais · Answer

For e g with ```aws ecr get login password region region | docker login username AWS password stdin <http aws account id dkr ecr region amazonaws com|aws account id dkr ecr region amazonaws com>```

Samuel Hinton · Answer

I assume that is indeed the issue the docker agent doesnt have an authentication step that I can see and the tokens you get from that last 12 hours only so it cant be something I run in the command manually before launching `prefect agent` otherwise the agent has to be restarted every twelve hours I am not sure how to tell the prefect agent Please use the credentials in your env variables to keep me logged into AWS so I can run flows

Jan Marais · Answer

Sorry I only use docker agents for debugging and never had the need for leaving them running gt 12 hours Maybe the information here will lead you to a solution <https docs aws amazon com AmazonECR latest userguide registry auth html>

Jan Marais · Answer

Note that simply providing an access key id and secret will not be enough to authenticate pull and push with docker and ECR

Samuel Hinton · Answer

Oh does it need something else

Jan Marais · Answer

See second paragraph of page following link above

Jan Marais · Answer

The Docker CLI doesn t support native IAM authentication methods Additional steps must be taken so that Amazon ECR can authenticate and authorize Docker push and pull requests

Samuel Hinton · Answer

Ah right gotcha Will hopefully see if someone else has managed to get this working within a DockerAgent

Karolína Bzdušek · Answer

This is just a thought not sure whether this would help right now I am trying to understand where and how to authenticate everything that it would run smoothly but except setting env variables have you share this in secrets in UI I have my GITHUB ACCES TOKEN exported in env variables but it seems that having share it as a secret in UI actually helps It s just a wild guess anyone who can enlighten me regarding authentication is very welcome

Samuel Hinton · Answer

I believe the issue is that the AWS authentication is something external to docker and thus harder to integrate The AWS cli needs to actually be called and passed to dockers auth and then for 12 hours Id get to utilise the ECR

Jan Marais · Answer

Maybe this will help <https github com awslabs amazon ecr credential helper> Seems like it will automatically fetch the credentials for ECR using your standard IAM credentials

Samuel Hinton · Answer

Ill see if I can get this working

Samuel Hinton · Answer

Cheers

Samuel Hinton · Answer

< Jan Marais> just wanted to chime back in and say this seems to help and I can see the docker agent Successfully pulled image 59630237498 It still doesnt run the flow but at least it now pulls the image down and starts the container

Jan Marais · Answer

Glad you are making progress Do you get any indication why the flow is not running

Samuel Hinton · Answer

Yes I believe there is a networking issue because the docker compose being run is actually part of a larger swarm which changes the network name and makes it unattachable Ill slap it into shape soon Im sure