@Marvin is there a way to prevent

prefect deploy --all

from pushing the docker image multiple times? my deploy has been getting

OSError: tag invalid: The image tag '49595c4' already exists in the 'dwh' repository and cannot be overwritten because the repository is immutable.

I'm tagging my docker image using the git commit hash

It seems there are no specific GitHub issues directly addressing this problem. However, the Prefect documentation provides some details about controlling Docker image builds and pushes. When using

prefect deploy

, you can customize the build and push process in the

prefect.yaml

file. This can help prevent multiple pushes of the same Docker image. Here are some relevant points and examples: 1. Use a Git Commit Hash for Tagging: You can set your Docker image's tag to a Git commit hash using a script. Here’s an example from the documentation:

yaml
   build:
     - prefect.deployments.steps.run_shell_script:
         id: get-commit-hash
         script: git rev-parse --short HEAD
         stream_output: false
     - prefect_docker.deployments.steps.build_docker_image:
         requires: prefect-docker
         image_name: my-image
         image_tag: "{{ get-commit-hash.stdout }}"
         dockerfile: auto

2. Deployment Instruction Overrides: You can override the

build

,

push

, and

pull

sections in the

prefect.yaml

file to cater to specific needs. This customization allows you to implement different behavior for different deployments, such as avoiding repeated Docker pushes:

yaml
   build:
     image_name: <your_image_name_here>
     image_tag: {{ your_commit_hash }}
     push: false  # or implement conditional logic

3. Customizing Push Logic: Consider implementing custom logic for pushing to handle immutable repositories correctly. You can add checks and conditions or use different strategies based on your needs:

yaml
   push:
     - prefect_shell_script:
         script: |
           if ! docker images -q <your_image_name>:{{ your_commit_hash }}; then
             docker push <your_image_name>:{{ your_commit_hash }}
           fi

Sources: - Prefect Deployment Guide - Prefect Kubernetes Deployment These references provide more specifics and could be instrumental in avoiding redundant image pushes.

cc @Nate I know you have thoughts on this

I saw this github issue that was related https://github.com/PrefectHQ/prefect/issues/9921

check this out: • define custom steps to cache image builds • use them

nice!!!

👍 ill just call out that step names like

prefect_docker.deployments.steps.build_docker_image

and the cached ones i have here that wrap those are just fully qualified names of python functions, so whatever python you have that runs

prefect deploy

will need access to those modules, which is why I have the steps defined in a package (but strictly speaking you shouldnt need a package)

does your custom step have an issue with

buildargs

? we use those to pass in github creds to install some private packages but I think it's causing some hangups due to an unhashable type dict.

aha - good point i didn't handle that but we should be able to handle that easily enough in the eventual version