Hey all! Having issues getting the DockerAgent attempting to do some DockerRun flows not being able to access the registry. We have a set of images in an Amazon ECR location, but I cant see any registry setting options in the DockerRun or DockerAgent (and having the

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

set doesnt seem to be what prefect is looking for), and it doesnt use them to authenticate. Does prefect only support public images?

Is your Docker client authenticated with ECR?

I assume that is indeed the issue - the docker agent doesnt have an authentication step that I can see, and the tokens you get from that last 12 hours only, so it cant be something I run in the command manually before launching

prefect agent

otherwise the agent has to be restarted every twelve hours. I am not sure how to tell the prefect agent “Please use the credentials in your env variables to keep me logged into AWS so I can run flows”

Sorry, I only use docker agents for debugging and never had the need for leaving them running > 12 hours. Maybe the information here will lead you to a solution: https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html

Oh, does it need something else?

See second paragraph of page following link above

Ah right, gotcha. Will hopefully see if someone else has managed to get this working within a DockerAgent

This is just a thought, not sure whether this would help (right now I am trying to understand where and how to authenticate everything that it would run smoothly), but except setting env variables, have you share this in secrets in UI? I have my GITHUB_ACCES_TOKEN exported in env variables but it seems that having share it as a secret in UI actually helps..It's just a wild guess, anyone who can enlighten me regarding authentication is very welcome.

I believe the issue is that the AWS authentication is something external to docker, and thus harder to integrate. The AWS cli needs to actually be called and passed to dockers auth, and then for 12 hours Id get to utilise the ECR

Maybe this will help: https://github.com/awslabs/amazon-ecr-credential-helper Seems like it will automatically fetch the credentials for ECR using your standard IAM credentials

Ill see if I can get this working

Glad you are making progress. Do you get any indication why the flow is not running?

Yes I believe there is a networking issue because the docker-compose being run is actually part of a larger swarm which changes the network name and makes it unattachable. Ill slap it into shape soon Im sure