@Marvin Can you write me the code for a prefect.yaml that authenticates with aws ECR and pulls in an image from ECR?

Ah, the joy of writing YAML files. It's like a never-ending existential crisis. But don't worry, I'm here to help you out. Here's a basic example of a

prefect.yaml

file that authenticates with AWS ECR and pulls an image:

yaml
version: 1
name: MyFlow
storage:
  type: Docker
  dockerfile: Dockerfile
  python_dependencies:
    - prefect
  image: <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com/<repository_name>:<tag|amazonaws.com/<repository_name>:<tag>>
  registry_url: <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
  base_image: python:3.8
  env_vars:
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
    AWS_SESSION_TOKEN: "${AWS_SESSION_TOKEN}"
  prefect_version: "0.14.0"
  python_version: "3.8"

Replace

<aws_account_id>

,

<region>

,

<repository_name>

, and

<tag>

with your AWS account ID, your AWS region, your ECR repository name, and your image tag, respectively. Remember to set your AWS credentials (

AWS_ACCESS_KEY_ID

,

AWS_SECRET_ACCESS_KEY

, and

AWS_SESSION_TOKEN

) as environment variables in the environment where you're running your Prefect agent. Keep in mind that this is a very basic example and may not cover all your needs. You might need to tweak it a bit to suit your specific requirements. But then again, isn't that the essence of programming? Constantly trying to fit square pegs into round holes. Ah, the joy.

@Nate Is there a good way to provide a step which injects aws creds from prefect blocks into the container pull step, so that a non-ecs container can access ECR and pull in the image?

hmm what do you mean by

the container pull step

I need to authenticate with aws ecr to get credentials in order for the container to pull in the ECR image to run the job. These creds expire, so it needs to happen before the image is pulled

aws ecr get-login-password --region <region> | docker login \ --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>

@Mitch @Nate I was just wondering the same thing, did you ever figure it out mithc?

Nope, I just switched to docker hub

@Mitch yeah, I was kind of thinking exactly this heh

sorry for not making it back to this thread - you'd auth in the worker here, not in the prefect.yaml, if im understanding correctly that you have some docker worker that needs to pull images from ECR

ECR auth is different as it requires auth every 12 hours

ah ok

@Nate it's really just a cli command one has to remember to run before running prefect deploy, that's the way I am doing it now

yeah i guess that assumes your runtime is free to pull images from that ECR already, which may be a good assumption depending on where you run your worker

If running in ECS it is easy as long as you have your roles configured correctly

yep - so i guess are you asking more about authing for push (deployment time) or authing for pull (run time)?

Mine was authing for a pull in a docker container not running in ECS

Yeah. Otherwise one can do this:

build:
- prefect.deployments.steps.run_shell_script:
    id: login-for-aws-ecr
    script: aws ecr get-login-password --region eu-north-1 | docker login --username AWS --password-stdin <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com|199214789681.dkr.ecr.eu-north-1.amazonaws.com>
    stream_output: true
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1 s3fs prefect-aws clickhouse-connect
    image_name: <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/ab-data-science-prefect-agent|199214789681.dkr.ecr.eu-north-1.amazonaws.com/ab-data-science-prefect-agent>
    tag: latest
    dockerfile: auto
    push: true
    platform: linux/amd64

What does push: true do in this scenario?