<@ULVA73B9P> the prefect deployment using docker b...
# ask-marvin
p
@Marvin the prefect deployment using docker build step is creating lots of stale images in my docker locally when i am deploying here is my yaml file - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.6.1 image_name: n0vum/soc-estimation_dresden-sued tag: '{{ get-soh-version.stdout }}' # The deployment needs to be run from the root directory of the project dockerfile: ./Dockerfile buildargs: GITLAB_ACCESS_TOKEN: "{{ prefect.blocks.secret.gitlab-token }}" network_mode: host nocache: True how to tackle this issue?
m
thought for 633.9 seconds
Thanks for sharing the step config — I see what’s happening. Why you’re seeing “stale” images: - You’re building a new image on every deploy with a unique tag (
'{{ get-soh-version.stdout }}'
). - You’re also forcing a no‑cache build (
nocache: True
), which guarantees new layers each time. - Docker does not garbage-collect old images/tags automatically, so they accumulate locally. Ways to tackle it 1) Stop forcing full rebuilds unless you need them - Remove
nocache: True
and let Docker layer cache work. In Prefect 3.x, the step parameter is
ignore_cache
(defaults to False), so just omit
nocache
entirely or set:
Copy code
ignore_cache: false
This will dramatically reduce redundant layers and build time. Docs: prefect-docker integration 2) Keep a stable local tag, but remove the versioned tag after push If you don’t need all historical images locally, keep just a stable tag like
latest
for local use and remove the versioned tag after you push. Example steps snippet:
Copy code
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
    requires: prefect-docker>=0.6.1
    image_name: n0vum/soc-estimation_dresden-sued
    tag: '{{ get-soh-version.stdout }}'
    additional_tags:
      - latest
    dockerfile: ./Dockerfile
    buildargs:
      GITLAB_ACCESS_TOKEN: "{{ prefect.blocks.secret.gitlab-token }}"
    network_mode: host
    ignore_cache: false

- prefect_docker.deployments.steps.push_docker_image:
    id: push_image
    image_name: n0vum/soc-estimation_dresden-sued
    tag: '{{ get-soh-version.stdout }}'

# Remove the just-built versioned tag locally to prevent piling up
- prefect_shell.commands.shell_run_command:
    id: cleanup_local_versioned_tag
    command: docker rmi n0vum/soc-estimation_dresden-sued:{{ get-soh-version.stdout }} || true
3) Periodic prune of unused images If you don’t run containers from these images locally, a prune step (either in your deploy or on a schedule) keeps Docker tidy: - Remove dangling layers:
Copy code
docker image prune -f
- Remove any images not used by containers and older than 24h:
Copy code
docker image prune -a -f --filter "until=24h"
You can run these with a small
prefect_shell.commands.shell_run_command
step. 4) Optional: prune by label (if you label your images) If your Dockerfile adds a label (e.g.,
LABEL com.prefect.deployment=my-deployment
), you can prune by label:
Copy code
docker images --filter "label=com.prefect.deployment=my-deployment" -q | tail -n +6 | xargs -r docker rmi
(This keeps the 5 most recent; adjust to taste.) References - Prefect YAML / steps docs: prefect.yaml guide - Docker integration: prefect-docker A couple quick questions to tailor the recommendation: - Are the “stale” ones dangling
<none>:<none>
images, or lots of old versioned tags (e.g.,
:1.2.3
,
:1.2.4
, etc.)? - Are you pushing to a registry as part of deploy, and do you need to keep multiple versions locally for testing?