https://prefect.io logo
m

Mitch

08/16/2023, 4:06 PM
@Marvin Can you write me the code for a prefect.yaml that authenticates with aws ECR and pulls in an image from ECR?
m

Marvin

08/16/2023, 4:07 PM
Ah, the joy of writing YAML files. It's like a never-ending existential crisis. But don't worry, I'm here to help you out. Here's a basic example of a
prefect.yaml
file that authenticates with AWS ECR and pulls an image:
Copy code
yaml
version: 1
name: MyFlow
storage:
  type: Docker
  dockerfile: Dockerfile
  python_dependencies:
    - prefect
  image: <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com/<repository_name>:<tag|amazonaws.com/<repository_name>:<tag>>
  registry_url: <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
  base_image: python:3.8
  env_vars:
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
    AWS_SESSION_TOKEN: "${AWS_SESSION_TOKEN}"
  prefect_version: "0.14.0"
  python_version: "3.8"
Replace
<aws_account_id>
,
<region>
,
<repository_name>
, and
<tag>
with your AWS account ID, your AWS region, your ECR repository name, and your image tag, respectively. Remember to set your AWS credentials (
AWS_ACCESS_KEY_ID
,
AWS_SECRET_ACCESS_KEY
, and
AWS_SESSION_TOKEN
) as environment variables in the environment where you're running your Prefect agent. Keep in mind that this is a very basic example and may not cover all your needs. You might need to tweak it a bit to suit your specific requirements. But then again, isn't that the essence of programming? Constantly trying to fit square pegs into round holes. Ah, the joy.
1
m

Mitch

08/16/2023, 9:06 PM
@Nate Is there a good way to provide a step which injects aws creds from prefect blocks into the container pull step, so that a non-ecs container can access ECR and pull in the image?
n

Nate

08/16/2023, 9:10 PM
hmm what do you mean by
the container pull step
m

Mitch

08/16/2023, 9:26 PM
I need to authenticate with aws ecr to get credentials in order for the container to pull in the ECR image to run the job. These creds expire, so it needs to happen before the image is pulled
Copy code
aws ecr get-login-password --region <region> | docker login \ --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
Essentially I need to pass in aws creds, call the above cli command which gets credentials for the container to be able to call
Copy code
docker pull <http://xxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/imagexyz|xxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/imagexyz>
m

Mattias Petter Johansson

09/14/2023, 4:30 PM
@Mitch @Nate I was just wondering the same thing, did you ever figure it out mithc?
m

Mitch

09/14/2023, 4:30 PM
Nope, I just switched to docker hub
m

Mattias Petter Johansson

09/14/2023, 4:31 PM
@Mitch yeah, I was kind of thinking exactly this heh
n

Nate

09/14/2023, 4:32 PM
sorry for not making it back to this thread - you'd auth in the worker here, not in the prefect.yaml, if im understanding correctly that you have some docker worker that needs to pull images from ECR
m

Mitch

09/14/2023, 4:33 PM
ECR auth is different as it requires auth every 12 hours
which is the issue here. So could auth the worker or run a script within the worker each time...
n

Nate

09/14/2023, 4:33 PM
ah ok
i know we can do this, i just havent had to do it myself, lemme try and find an example\
m

Mattias Petter Johansson

09/14/2023, 4:42 PM
@Nate it's really just a cli command one has to remember to run before running prefect deploy, that's the way I am doing it now
n

Nate

09/14/2023, 4:43 PM
yeah i guess that assumes your runtime is free to pull images from that ECR already, which may be a good assumption depending on where you run your worker
m

Mitch

09/14/2023, 4:44 PM
If running in ECS it is easy as long as you have your roles configured correctly
n

Nate

09/14/2023, 4:44 PM
yep - so i guess are you asking more about authing for push (deployment time) or authing for pull (run time)?
m

Mitch

09/14/2023, 4:45 PM
Mine was authing for a pull in a docker container not running in ECS
m

Mattias Petter Johansson

09/14/2023, 5:02 PM
Yeah. Otherwise one can do this:
Copy code
build:
- prefect.deployments.steps.run_shell_script:
    id: login-for-aws-ecr
    script: aws ecr get-login-password --region eu-north-1 | docker login --username AWS --password-stdin <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com|199214789681.dkr.ecr.eu-north-1.amazonaws.com>
    stream_output: true
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.3.1 s3fs prefect-aws clickhouse-connect
    image_name: <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/ab-data-science-prefect-agent|199214789681.dkr.ecr.eu-north-1.amazonaws.com/ab-data-science-prefect-agent>
    tag: latest
    dockerfile: auto
    push: true
    platform: linux/amd64
👀 1
m

Mitch

09/14/2023, 5:58 PM
What does push: true do in this scenario?