https://prefect.io logo
s

Samuel Hinton

02/10/2021, 1:15 PM
Hey all! Having issues getting the DockerAgent attempting to do some DockerRun flows not being able to access the registry. We have a set of images in an Amazon ECR location, but I cant see any registry setting options in the DockerRun or DockerAgent (and having the
AWS_ACCESS_KEY_ID
and
AWS_SECRET_ACCESS_KEY
set doesnt seem to be what prefect is looking for), and it doesnt use them to authenticate. Does prefect only support public images?
j

Jan Marais

02/10/2021, 1:43 PM
Is your Docker client authenticated with ECR?
For e.g. with:
Copy code
aws ecr get-login-password --region region | docker login --username AWS --password-stdin <http://aws_account_id.dkr.ecr.region.amazonaws.com|aws_account_id.dkr.ecr.region.amazonaws.com>
s

Samuel Hinton

02/10/2021, 1:45 PM
I assume that is indeed the issue - the docker agent doesnt have an authentication step that I can see, and the tokens you get from that last 12 hours only, so it cant be something I run in the command manually before launching
prefect agent
otherwise the agent has to be restarted every twelve hours. I am not sure how to tell the prefect agent “Please use the credentials in your env variables to keep me logged into AWS so I can run flows”
j

Jan Marais

02/10/2021, 1:51 PM
Sorry, I only use docker agents for debugging and never had the need for leaving them running > 12 hours. Maybe the information here will lead you to a solution: https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html
Note that simply providing an access key id and secret will not be enough to authenticate pull and push with docker and ECR
s

Samuel Hinton

02/10/2021, 1:53 PM
Oh, does it need something else?
j

Jan Marais

02/10/2021, 1:54 PM
See second paragraph of page following link above
"The Docker CLI doesn't support native IAM authentication methods. Additional steps must be taken so that Amazon ECR can authenticate and authorize Docker push and pull requests."
s

Samuel Hinton

02/10/2021, 1:55 PM
Ah right, gotcha. Will hopefully see if someone else has managed to get this working within a DockerAgent
k

Karolína Bzdušek

02/10/2021, 2:01 PM
This is just a thought, not sure whether this would help (right now I am trying to understand where and how to authenticate everything that it would run smoothly), but except setting env variables, have you share this in secrets in UI? I have my GITHUB_ACCES_TOKEN exported in env variables but it seems that having share it as a secret in UI actually helps..It's just a wild guess, anyone who can enlighten me regarding authentication is very welcome.
s

Samuel Hinton

02/10/2021, 2:02 PM
I believe the issue is that the AWS authentication is something external to docker, and thus harder to integrate. The AWS cli needs to actually be called and passed to dockers auth, and then for 12 hours Id get to utilise the ECR
j

Jan Marais

02/10/2021, 2:09 PM
Maybe this will help: https://github.com/awslabs/amazon-ecr-credential-helper Seems like it will automatically fetch the credentials for ECR using your standard IAM credentials
🙏 1
s

Samuel Hinton

02/10/2021, 2:11 PM
Ill see if I can get this working
Cheers!
@Jan Marais just wanted to chime back in and say this seems to help, and I can see the docker agent “Successfully pulled image 59630237498……“. It still doesnt run the flow, but at least it now pulls the image down and starts the container!
j

Jan Marais

02/10/2021, 6:00 PM
Glad you are making progress. Do you get any indication why the flow is not running?
s

Samuel Hinton

02/10/2021, 7:09 PM
Yes I believe there is a networking issue because the docker-compose being run is actually part of a larger swarm which changes the network name and makes it unattachable. Ill slap it into shape soon Im sure