<@ULVA73B9P> how to just build a flow in CI and de...
# marvin-ai
l
@Marvin how to just build a flow in CI and deploy later at publish time to kubernetes ?
m
thought for 432.4 seconds
Great question — in Prefect 3.x you can (and should) decouple build from deploy. The common pattern is: - Build a container image with your code/deps in CI and push it to your registry (no Prefect action here). - Later (at “publish” time), create/update the Prefect deployment to point at that image and a Kubernetes work pool — with build/push disabled. Two common approaches for source code: A) Git-based source (recommended for decoupling code from image) - The image just provides the runtime environment; the worker clones your repo at run time. - Deployment references your Git repo and entrypoint. B) Image-baked source (bake your code into the image) - The image contains your code and dependencies; deployment uses that image and entrypoint path. Below are minimal examples for both. One-time setup (Kubernetes work pool) - Create a Kubernetes work pool once:
Copy code
prefect work-pool create my-k8s --type kubernetes
- Ensure a worker is running for that pool (wherever it has K8s permissions):
Copy code
prefect worker start -p my-k8s
CI step (build only) - Example (GitHub Actions, GitLab, etc.):
Copy code
docker build -t <http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}> .
docker push <http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}>
Publish step (deploy only) Option A: Git-based source (workers clone your repo) - Use a small deploy script so you can pass the exact image tag created in CI. This avoids deprecated 2.x commands.
Copy code
# deploy.py
import os
from prefect.flow import Flow

IMAGE = os.environ["IMAGE"]  # e.g. <http://ghcr.io/acme/prefect-flows:abcdef|ghcr.io/acme/prefect-flows:abcdef>

Flow.from_source(
    "<https://github.com/acme/data-pipelines>",  # your repo
    entrypoint="flows/my_flow.py:my_flow",     # path in repo to the flow
).deploy(
    name="my-flow-prod",
    work_pool_name="my-k8s",
    image=IMAGE,
    build=False,   # IMPORTANT: do not build here
    push=False,    # IMPORTANT: do not push here
)
- Then run at publish time:
Copy code
IMAGE=<http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}> python deploy.py
Option B: Image-baked source (code is inside the image) - Import your flow and deploy with the prebuilt image:
Copy code
# deploy.py
import os
from flows.my_flow import my_flow  # local import of your flow

IMAGE = os.environ["IMAGE"]

my_flow.deploy(
    name="my-flow-prod",
    work_pool_name="my-k8s",
    image=IMAGE,
    build=False,
    push=False,
)
- Run at publish time:
Copy code
IMAGE=<http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}> python deploy.py
Using prefect.yaml (optional) - If you prefer CLI + prefect.yaml, commit a prefect.yaml that sets the work pool and entrypoint; at publish time you can set the image via job variables or flags:
Copy code
prefect deploy ./flows/my_flow.py:my_flow --name my-flow-prod --work-pool my-k8s
- Then in your prefect.yaml, set the Kubernetes job variables (like image) or pass them in via environment at deploy time. If you prefer full control in code, the Python deploy() examples above are simpler and explicit. Important notes - In Prefect 3.x, do not use removed 2.x commands like: - Deployment.build_from_flow() - prefect deployment build - GitHub storage blocks - Instead use Flow.from_source(...).deploy(...), or import the flow and call .deploy(...) directly. - Make sure your Kubernetes work pool has permissions and a running worker. - To pin a specific revision with the Git-based approach, deploy from the commit you want or include your ref in the source URL and ensure the worker has credentials to clone. Helpful links - Kubernetes work pools (how-to): Prefect 3 docs: Kubernetes work pools - Flow deployment with images and work pools: Prefect blog: Work pools and workers - Docker utils (if you choose to build images via Python): prefect.utilities.dockerutils API
Would you like to use the Git-based runtime source or bake your code into the image? I can tailor the exact deploy script or prefect.yaml to your repo layout and registry.
l
@Marvin how to build using perfect api
m
thought for 94.7 seconds
If you want to “build in CI using the Prefect API”, use Prefect’s docker helpers. In Prefect 3.x, the
prefect.utilities.dockerutils
module exposes simple functions to build and push images. Requirements - Ensure Docker is available in your CI runner. - Install Prefect with Docker extras:
pip install 'prefect[docker]'
(or run your script via
uv run --with 'prefect[docker]' ...
). - Authenticate your CI to your registry (e.g., GHCR/ECR/GCR). Minimal build (existing Dockerfile)
Copy code
from pathlib import Path
from prefect.utilities.dockerutils import build_image, push_image

REGISTRY = "<http://ghcr.io|ghcr.io>"
REPO = "acme/prefect-flows"
TAG = "abcdef"  # e.g., your commit SHA
FULL_TAG = f"{REGISTRY}/{REPO}:{TAG}"

# Build using your Dockerfile in the current directory
image_id = build_image(
    context=Path("."),
    dockerfile="Dockerfile",
    tag=FULL_TAG,        # optional; tags the built image
    pull=True,           # pull newer base layers
    platform="linux/amd64",  # optional; requires buildx/daemon support
)

# Optionally push using Prefect's API
push_image(
    image_id=image_id,
    registry_url=REGISTRY,
    name=REPO,
    tag=TAG,
)

print(f"Built and pushed: {FULL_TAG}")
No Dockerfile? Generate a sane default, then build
Copy code
from pathlib import Path
from prefect.utilities.dockerutils import generate_default_dockerfile, build_image, push_image

dockerfile_text = generate_default_dockerfile()  # returns Dockerfile contents
Path("Dockerfile").write_text(dockerfile_text)

# Now build and push as above
A small helper you can drop into CI
Copy code
from pathlib import Path
from prefect.utilities.dockerutils import build_image, push_image

def build_and_push(image_repo: str, tag: str, context: str=".", dockerfile: str="Dockerfile", platform: str | None=None) -> str:
    full = f"{image_repo}:{tag}"
    image_id = build_image(
        context=Path(context),
        dockerfile=dockerfile,
        tag=full,
        pull=True,
        platform=platform,
    )
    registry_url, name = image_repo.split("/", 1)  # e.g., "<http://ghcr.io|ghcr.io>", "acme/prefect-flows"
    push_image(image_id=image_id, registry_url=registry_url, name=name, tag=tag)
    return full

# Example
# built = build_and_push("<http://ghcr.io/acme/prefect-flows|ghcr.io/acme/prefect-flows>", os.environ["GIT_SHA"], platform="linux/amd64")
Next step: deploy later - At publish time, point your deployment to this prebuilt image and disable build/push:
Copy code
from prefect.flow import Flow

Flow.from_source(
    "<https://github.com/acme/data-pipelines>",
    entrypoint="flows/my_flow.py:my_flow",
).deploy(
    name="my-flow-prod",
    work_pool_name="my-k8s",
    image="<http://ghcr.io/acme/prefect-flows:abcdef|ghcr.io/acme/prefect-flows:abcdef>",
    build=False,
    push=False,
)
Docs - prefect.utilities.dockerutils API If you share your registry and tag convention, I can tailor the build script to your CI.