<@ULVA73B9P> During CI/CD I am deploying deploymen...
# ask-marvin
m
@Marvin During CI/CD I am deploying deployments. Is there a way to no re-build the same docker image multiple times if multiple deployments use the same dockerfile
m
thought for 323.3 seconds
Yes. In Prefect 3.x you can build/push a single image once and have multiple deployments reference it, so CI/CD doesn’t rebuild it for each deployment. Common patterns: 1) Build once in prefect.yaml and reuse across deployments - Put a single docker build/push step at the top of your project’s
prefect.yaml
- Reference the image from each deployment’s job variables
Copy code
# prefect.yaml
name: my-project

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.3.1
      image_name: my-registry/my-image
      tag: "{{ prefect.deployments.work_pool_name | default('ci') }}-{{ prefect.deployments.name }}"
      dockerfile: ./Dockerfile

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.3.1
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"

deployments:
  - name: ingest
    entrypoint: flows/ingest.py:ingest
    work_pool:
      name: docker-pool
      job_variables:
        image: "{{ build-image.image }}"

  - name: transform
    entrypoint: flows/transform.py:transform
    work_pool:
      name: docker-pool
      job_variables:
        image: "{{ build-image.image }}"
Then run:
Copy code
prefect deploy --all
The build/push runs once; both deployments use the same image. Tip: Use a stable tag like your commit SHA so all deployments in that CI run point to the same image: - Pre-step to capture
git rev-parse --short HEAD
- Set
tag: "{{ get_commit.stdout }}"
2) Image is prebuilt by CI (skip Prefect build entirely) If you build the image in an earlier CI job, skip building in Prefect and just point deployments at the existing tag:
Copy code
# prefect.yaml
build: null
push: null

deployments:
  - name: ingest
    entrypoint: flows/ingest.py:ingest
    work_pool:
      name: docker-pool
      job_variables:
        image: "my-registry/my-image:${GIT_SHA}"

  - name: transform
    entrypoint: flows/transform.py:transform
    work_pool:
      name: docker-pool
      job_variables:
        image: "my-registry/my-image:${GIT_SHA}"
Or via Python:
Copy code
from prefect import flow

@flow
def my_flow(): ...

my_flow.deploy(
    name="ingest",
    work_pool_name="docker-pool",
    image="my-registry/my-image:sha-1234",
    build=False,
    push=False,
)
Notes - Within a single
prefect deploy --all
run, the build step executes once; all deployments can reference its output. - To control Docker layer caching during builds, pass args in the build step (e.g.,
pull: true
,
nocache: false
,
buildargs
,
platform
). - This works the same whether your worker runs Docker, Kubernetes, ECS, etc.—each uses the same image reference in
job_variables.image
. Docs - prefect.yaml steps and templating - Create deployments (CLI) - Deploy via Python API - Customize job variables - prefect-docker integration If you can share how you’re building today (prefect.yaml vs Python API) and which worker type you’re targeting (Docker/Kubernetes/etc.), I can tailor an example to your setup.
m
@Marvin I have environment specific deployments (dev/staging/prod). Is it possible to group deployments? Can't really leverage --all
m
thought for 47.4 seconds
Yes—use naming conventions plus the
--name
pattern filter, or split configs per env. There isn’t a first-class “group” concept, but these patterns work well in CI. Options: 1) Name-based groups with
--name
- Give each deployment an env-specific name (e.g.,
dev
,
staging
,
prod
) and select with wildcards. Examples:
Copy code
# deploy all dev across all flows in the project
prefect deploy -n "*/dev"

# deploy only prod deployments (deployment names starting with 'prod-')
prefect deploy -n "*/*-prod*"

# deploy dev for a specific flow only
prefect deploy -n "orders/dev"
Tip: quote patterns so your shell doesn’t expand them. 2) Separate prefect.yaml files per environment - Keep environment-specific deployments in separate files and call the right one in CI:
Copy code
prefect deploy --prefect-file prefect.dev.yaml
prefect deploy --prefect-file prefect.staging.yaml
prefect deploy --prefect-file prefect.prod.yaml
3) Programmatic deploy (Python) - If you prefer code, a tiny script can deploy just one env’s set (handy when your logic gets more complex). Notes - The
--all
flag deploys everything; for env subsets, prefer
-n/--name
patterns or separate files. - There’s no CLI filter by tags for creation; selection is by name pattern or file. Docs - Create deployments (CLI) - prefect.yaml structure If you share how your deployments are currently named, I can suggest exact patterns (e.g., whether you use
dev-foo
,
foo-dev
, or
dev
as the deployment name).