Mitch
08/16/2023, 4:06 PMMarvin
08/16/2023, 4:07 PMprefect.yaml
file that authenticates with AWS ECR and pulls an image:
yaml
version: 1
name: MyFlow
storage:
type: Docker
dockerfile: Dockerfile
python_dependencies:
- prefect
image: <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com/<repository_name>:<tag|amazonaws.com/<repository_name>:<tag>>
registry_url: <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
base_image: python:3.8
env_vars:
AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
AWS_SESSION_TOKEN: "${AWS_SESSION_TOKEN}"
prefect_version: "0.14.0"
python_version: "3.8"
Replace <aws_account_id>
, <region>
, <repository_name>
, and <tag>
with your AWS account ID, your AWS region, your ECR repository name, and your image tag, respectively.
Remember to set your AWS credentials (AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
, and AWS_SESSION_TOKEN
) as environment variables in the environment where you're running your Prefect agent.
Keep in mind that this is a very basic example and may not cover all your needs. You might need to tweak it a bit to suit your specific requirements. But then again, isn't that the essence of programming? Constantly trying to fit square pegs into round holes. Ah, the joy.Mitch
08/16/2023, 9:06 PMNate
08/16/2023, 9:10 PMthe container pull step
Mitch
08/16/2023, 9:26 PMaws ecr get-login-password --region <region> | docker login \ --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
docker pull <http://xxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/imagexyz|xxxxxxxxxx.dkr.ecr.us-east-2.amazonaws.com/imagexyz>
Mattias Petter Johansson
09/14/2023, 4:30 PMMitch
09/14/2023, 4:30 PMMattias Petter Johansson
09/14/2023, 4:31 PMNate
09/14/2023, 4:32 PMMitch
09/14/2023, 4:33 PMNate
09/14/2023, 4:33 PMMattias Petter Johansson
09/14/2023, 4:42 PMNate
09/14/2023, 4:43 PMMitch
09/14/2023, 4:44 PMNate
09/14/2023, 4:44 PMMitch
09/14/2023, 4:45 PMMattias Petter Johansson
09/14/2023, 5:02 PMbuild:
- prefect.deployments.steps.run_shell_script:
id: login-for-aws-ecr
script: aws ecr get-login-password --region eu-north-1 | docker login --username AWS --password-stdin <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com|199214789681.dkr.ecr.eu-north-1.amazonaws.com>
stream_output: true
- prefect_docker.deployments.steps.build_docker_image:
id: build_image
requires: prefect-docker>=0.3.1 s3fs prefect-aws clickhouse-connect
image_name: <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/ab-data-science-prefect-agent|199214789681.dkr.ecr.eu-north-1.amazonaws.com/ab-data-science-prefect-agent>
tag: latest
dockerfile: auto
push: true
platform: linux/amd64
Mitch
09/14/2023, 5:58 PM