lucian
09/19/2025, 8:44 AMMarvin
09/19/2025, 8:44 AMMarvin
09/19/2025, 8:51 AMprefect work-pool create my-k8s --type kubernetes
- Ensure a worker is running for that pool (wherever it has K8s permissions):
prefect worker start -p my-k8s
CI step (build only)
- Example (GitHub Actions, GitLab, etc.):
docker build -t <http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}> .
docker push <http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}>
Publish step (deploy only)
Option A: Git-based source (workers clone your repo)
- Use a small deploy script so you can pass the exact image tag created in CI. This avoids deprecated 2.x commands.
# deploy.py
import os
from prefect.flow import Flow
IMAGE = os.environ["IMAGE"] # e.g. <http://ghcr.io/acme/prefect-flows:abcdef|ghcr.io/acme/prefect-flows:abcdef>
Flow.from_source(
"<https://github.com/acme/data-pipelines>", # your repo
entrypoint="flows/my_flow.py:my_flow", # path in repo to the flow
).deploy(
name="my-flow-prod",
work_pool_name="my-k8s",
image=IMAGE,
build=False, # IMPORTANT: do not build here
push=False, # IMPORTANT: do not push here
)
- Then run at publish time:
IMAGE=<http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}> python deploy.py
Option B: Image-baked source (code is inside the image)
- Import your flow and deploy with the prebuilt image:
# deploy.py
import os
from flows.my_flow import my_flow # local import of your flow
IMAGE = os.environ["IMAGE"]
my_flow.deploy(
name="my-flow-prod",
work_pool_name="my-k8s",
image=IMAGE,
build=False,
push=False,
)
- Run at publish time:
IMAGE=<http://ghcr.io/acme/prefect-flows:${GIT_SHA}|ghcr.io/acme/prefect-flows:${GIT_SHA}> python deploy.py
Using prefect.yaml (optional)
- If you prefer CLI + prefect.yaml, commit a prefect.yaml that sets the work pool and entrypoint; at publish time you can set the image via job variables or flags:
prefect deploy ./flows/my_flow.py:my_flow --name my-flow-prod --work-pool my-k8s
- Then in your prefect.yaml, set the Kubernetes job variables (like image) or pass them in via environment at deploy time. If you prefer full control in code, the Python deploy() examples above are simpler and explicit.
Important notes
- In Prefect 3.x, do not use removed 2.x commands like:
- Deployment.build_from_flow()
- prefect deployment build
- GitHub storage blocks
- Instead use Flow.from_source(...).deploy(...), or import the flow and call .deploy(...) directly.
- Make sure your Kubernetes work pool has permissions and a running worker.
- To pin a specific revision with the Git-based approach, deploy from the commit you want or include your ref in the source URL and ensure the worker has credentials to clone.
Helpful links
- Kubernetes work pools (how-to): Prefect 3 docs: Kubernetes work pools
- Flow deployment with images and work pools: Prefect blog: Work pools and workers
- Docker utils (if you choose to build images via Python): prefect.utilities.dockerutils APIMarvin
09/19/2025, 8:51 AMlucian
09/19/2025, 8:57 AMMarvin
09/19/2025, 8:57 AMMarvin
09/19/2025, 8:58 AMprefect.utilities.dockerutils
module exposes simple functions to build and push images.
Requirements
- Ensure Docker is available in your CI runner.
- Install Prefect with Docker extras: pip install 'prefect[docker]'
(or run your script via uv run --with 'prefect[docker]' ...
).
- Authenticate your CI to your registry (e.g., GHCR/ECR/GCR).
Minimal build (existing Dockerfile)
from pathlib import Path
from prefect.utilities.dockerutils import build_image, push_image
REGISTRY = "<http://ghcr.io|ghcr.io>"
REPO = "acme/prefect-flows"
TAG = "abcdef" # e.g., your commit SHA
FULL_TAG = f"{REGISTRY}/{REPO}:{TAG}"
# Build using your Dockerfile in the current directory
image_id = build_image(
context=Path("."),
dockerfile="Dockerfile",
tag=FULL_TAG, # optional; tags the built image
pull=True, # pull newer base layers
platform="linux/amd64", # optional; requires buildx/daemon support
)
# Optionally push using Prefect's API
push_image(
image_id=image_id,
registry_url=REGISTRY,
name=REPO,
tag=TAG,
)
print(f"Built and pushed: {FULL_TAG}")
No Dockerfile? Generate a sane default, then build
from pathlib import Path
from prefect.utilities.dockerutils import generate_default_dockerfile, build_image, push_image
dockerfile_text = generate_default_dockerfile() # returns Dockerfile contents
Path("Dockerfile").write_text(dockerfile_text)
# Now build and push as above
A small helper you can drop into CI
from pathlib import Path
from prefect.utilities.dockerutils import build_image, push_image
def build_and_push(image_repo: str, tag: str, context: str=".", dockerfile: str="Dockerfile", platform: str | None=None) -> str:
full = f"{image_repo}:{tag}"
image_id = build_image(
context=Path(context),
dockerfile=dockerfile,
tag=full,
pull=True,
platform=platform,
)
registry_url, name = image_repo.split("/", 1) # e.g., "<http://ghcr.io|ghcr.io>", "acme/prefect-flows"
push_image(image_id=image_id, registry_url=registry_url, name=name, tag=tag)
return full
# Example
# built = build_and_push("<http://ghcr.io/acme/prefect-flows|ghcr.io/acme/prefect-flows>", os.environ["GIT_SHA"], platform="linux/amd64")
Next step: deploy later
- At publish time, point your deployment to this prebuilt image and disable build/push:
from prefect.flow import Flow
Flow.from_source(
"<https://github.com/acme/data-pipelines>",
entrypoint="flows/my_flow.py:my_flow",
).deploy(
name="my-flow-prod",
work_pool_name="my-k8s",
image="<http://ghcr.io/acme/prefect-flows:abcdef|ghcr.io/acme/prefect-flows:abcdef>",
build=False,
push=False,
)
Docs
- prefect.utilities.dockerutils API
If you share your registry and tag convention, I can tailor the build script to your CI.