Great questions — here’s how to set up a Kubernetes worker with Helm and use your custom flow image in Prefect 3.x. High-level - The Helm release for prefect-worker runs the worker process only. Your custom Docker image (with your flows + libs) should be used by the flow run pods, not by the worker pod. - You set the flow run image in the work pool (base job template) or per deployment (job_variables). Don’t change the worker image to your flow image unless you have a specific reason. - Deployments are created with prefect deploy (or .deploy in code) and point to your Kubernetes work pool. - Custom blocks are registered once against your server; your runtime image should include the Python code for any custom block classes you import in flows. 1) Install the Kubernetes worker with the official Helm chart - Helm repo: prefect-helm - Worker chart: prefect-worker Example minimal values.yaml for a self-hosted Prefect Server - Replace the API URL with your in-cluster DNS for the server (common pattern shown). If you enabled basic auth on the server chart, include those credentials; otherwise you can omit auth. - Set the work pool name to the Kubernetes pool you’ll use for deployments.

worker:
  # Use self-hosted server API (you installed prefect-server via Helm)
  apiConfig: selfHostedServer
  selfHostedServerApiConfig:
    apiUrl: <http://prefect-server.prefect.svc.cluster.local:4200/api>
    basicAuth:
      enabled: false

  # Worker process configuration
  config:
    workPool: my-k8s-pool         # REQUIRED: name of your Kubernetes work pool
    prefetchSeconds: 10
    queryInterval: 5

image:
  repository: prefecthq/prefect
  tag: 3-python3.11-kubernetes     # includes prefect-kubernetes integration
  pullPolicy: IfNotPresent

# Service account / RBAC for creating Jobs in the target namespace
serviceAccount:
  create: true

rbac:
  create: true

# Optional: only affects pulling the WORKER image, not your flow-run image
imagePullSecrets: []
# - name: my-registry-secret

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 1000m
    memory: 1Gi

Install

helm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update
helm install prefect-worker prefect/prefect-worker -n prefect -f values.yaml

Links: - Chart repo: prefect-helm - Worker chart readme: prefect-worker - Kubernetes how-to: Kubernetes worker guide 2) Where to set your custom flow image (the one with your code) Do not put your flow image on the worker pod. Instead, set it as the image for the flow-run Jobs: - Option A: Set default image at the work pool level (UI) - Work Pools → your Kubernetes pool → Configuration → Image =

my-registry/my-flows:tag

- Option B: Set per-deployment via job_variables - prefect.yaml:

deployments:
      - name: my-deployment
        entrypoint: flows/my_flow.py:my_flow
        work_pool:
          name: my-k8s-pool
          job_variables:
            image: my-registry/my-flows:tag
            # if your flow image is in a private registry, add pull secrets here too
            image_pull_secrets:
              - my-registry-secret

- Or in Python: ``` from prefect import flow @flow def my_flow(): ... if name == "__main__": my_flow.from_source( source="https://github.com/my-org/my-repo.git", entrypoint="flows/my_flow.py:my_flow", ).deploy( name="my-deployment", work_pool_name="my-k8s-pool", job_variables={ "image": "my-registry/my-flows:tag", "image_pull_secrets": ["my-registry-secret"], }, )

- Option C: Customize the work pool’s base job template
  - You can set a default `image` and `image_pull_secrets` in the pool’s base job template so all deployments inherit them.
  - Docs: <https://docs.prefect.io/v3/advanced/customize-base-job-templates|Customize base job templates>

Important:
- imagePullSecrets in your Helm values only affect the worker deployment. To let flow-run pods pull from a private registry, specify image_pull_secrets in the work pool base job template or in each deployment’s job_variables.

3) Deploying flows (Prefect 3.x)
- Prefer the prefect.yaml + `prefect deploy` workflow.
- Validate the CLI exists and syntax is correct (checked): `prefect deploy --help`
- Example minimal prefect.yaml for your “code baked into image” pattern:

deployments: - name: my-deployment entrypoint: flows/my_flow.py:my_flow work_pool: name: my-k8s-pool job_variables: image: my-registry/my-flows:tag image_pull_secrets: - my-registry-secret

- Then run:

prefect deploy

- At runtime, Prefect imports `flows/my_flow.py:my_flow` inside your image; ensure that path exists in the container.

Note on source vs image:
- Since you’ve baked code into the image, you don’t need a git clone “pull” step.
- If you prefer to keep a slim image and pull code at runtime, use the git steps in prefect.yaml instead. Docs: <https://docs.prefect.io/v3/how-to-guides/deployment_infra/kubernetes|Kubernetes guide>

4) Custom blocks (creating and using)
- Define your block classes once and register them against your server:

prefect block register --file path/to/my_blocks.py

- Or register a module:

prefect block register --module my_package.blocks

- Create block instances via the UI or Python and save them.
- If your flows import your custom block classes, ensure the runtime image includes those modules so `from my_blocks import MyBlock` works at run time.

Docs:
- Blocks overview: <https://docs.prefect.io/latest/concepts/blocks|Blocks concept>
- Kubernetes integration API (built-in blocks): <https://docs.prefect.io/integrations/prefect-kubernetes/api-ref/prefect_kubernetes-worker|prefect-kubernetes API ref>

5) Creating/starting workers
- With Helm, you do not run `prefect worker start` yourself. The Helm chart deploys a Kubernetes Deployment that runs the worker process and polls your specified work pool.
- If you want multiple workers/HA, scale the Helm deployment (replicaCount or HPA).
- For development, you can still run a local worker:

prefect work-pool create "my-k8s-pool" --type kubernetes # if you haven’t already prefect worker start --pool "my-k8s-pool" ``` But in-cluster you typically rely on the Helm-managed worker. Common gotchas - Jobs don’t start: ensure the work pool name in Helm values matches the pool where you deploy, and that RBAC allows creating Jobs in the target namespace. - Image pull errors: make sure

image_pull_secrets

are set in the job template or deployment job_variables — not just on the worker. - API URL: for self-hosted, use the in-cluster service URL ending with

/api

(e.g.,

http://<release>-server.<ns>.svc.cluster.local:4200/api

). If you share: - the namespace you deployed

prefect-server

into, - whether you enabled basic auth on the server, - your desired work pool name and target namespaces for jobs, I can draft an exact values.yaml and a base job template for your setup.

Great questions — here are crisp answers for each: 1) Where does the flow image live and when is it pulled? - The image that contains Python, libraries, and your flow code is used by the flow-run Job pods, not the worker pod. You reference it in the work pool’s base job template or per-deployment via

job_variables

in

prefect.yaml

. - Kubernetes pulls that image when a flow-run Job starts, not when you run

prefect deploy

. - Pull behavior is standard Kubernetes: - If

imagePullPolicy: IfNotPresent

(Prefect’s default in the base job template), the node will reuse a cached image if it already has that exact tag. - If

imagePullPolicy: Always

, the node will pull on every run. - Best practice: use immutable tags (e.g.,

my-flows:2026-04-07-1

) and keep

IfNotPresent

so nodes only pull when the tag changes or the node hasn’t cached it yet. Helpful docs: - Work pools and job templates: Customize base job templates - Kubernetes how-to: Kubernetes worker guide 2) What is the prefect-worker pod actually doing? - The worker polls your Prefect API for scheduled runs in its work pool. - For each run, it renders your work pool’s base job template + deployment job_variables and creates a Kubernetes Job to execute the flow. - It streams logs, emits events, observes pod/job status, and reports state back to Prefect. It also handles API key propagation to jobs (depending on your chart settings). - The worker does not execute your flow code itself; it’s the controller/orchestrator that submits and monitors the Kubernetes Jobs. Chart links: - Helm repo: prefect-helm - Worker chart: prefect-worker 3) Updating blocks/flow code and rolling out a new image — what’s the process? - You do not run

prefect deploy

in the worker pod. Treat the worker as a controller only. Build/publish and deploy from CI or your workstation. - Typical flow (image-baked code): - Build and push a new image tag with your updated flow code and any updated block class definitions. - Point deployments to the new tag: - If you set the image per-deployment: update

prefect.yaml

job_variables (or use

.deploy(..., job_variables={"image": ...})

) and run

prefect deploy

. - If you set a default image in the work pool’s base job template: update it in the UI (or via API). No

prefect deploy

needed; new flow runs will use the updated template. - Block updates: - If you changed only a block instance value (like credentials), update it in the UI or via Python. No redeploy needed; flows that

load()

the block will see the new value on next run. - If you changed a block type’s schema/class, re-register it once:

prefect block register --file path/to/my_blocks.py

Ensure your runtime image includes the updated module so imports succeed in the flow-run pod. - You only need to redeploy (run

prefect deploy

) when you want to: - Create a new deployment, or - Update deployment metadata (parameters, schedules, work pool, job_variables, etc.) - You do not need to restart the worker to pick up new pool templates or deployments; it continuously pulls configuration from the API. Only Helm-level changes to the worker itself (e.g., API URL, work pool name, RBAC) require a

helm upgrade

. Quick examples - Per-deployment image (with private registry secret for flow-run pods):

deployments:
  - name: my-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-k8s-pool
      job_variables:
        image: my-registry/my-flows:2026-04-07-1
        image_pull_secrets:
          - my-registry-secret

- Create a Kubernetes work pool (once):

prefect work-pool create "my-k8s-pool" --type kubernetes

If you’d like, share:

- your work pool name, - whether your flow image is in a private registry, - and if you prefer pool-level default or per-deployment override, and I’ll tailor a ready-to-use

prefect.yaml

and, if needed, a base job template JSON for your setup.

Great question. In Prefect 3.x, the best place to define environment variables for your flows depends on how you run them: - If you deploy to a work pool (Docker, Kubernetes, etc.) — recommended - Define them on the deployment via

job_variables.env

. These are injected into the flow run’s container/process and override pool defaults. - Example (Python):

from prefect import flow

    @flow
    def my_flow():
        import os
        print(os.environ.get("MLFLOW_TRACKING_URI"))

    if __name__ == "__main__":
        my_flow.deploy(
            name="mlflow-prod",
            work_pool_name="my-docker-pool",
            image="my/image:latest",
            job_variables={
                "env": {
                    "MLFLOW_TRACKING_URI": "<http://mlflow.prod:5000>",
                    "MLFLOW_EXPERIMENT_NAME": "production"
                }
            },
        )

- Example (prefect.yaml):

deployments:
      - name: mlflow-prod
        entrypoint: flows/train.py:train
        work_pool:
          name: my-docker-pool
          job_variables:
            env:
              MLFLOW_TRACKING_URI: "<http://mlflow.prod:5000>"
              MLFLOW_EXPERIMENT_NAME: "production"

- Docs: Deployments, Customize job variables - If you want defaults for every deployment in a pool - Put them in the work pool’s base job template (Environment Variables section). Deployment-level

job_variables.env

can override these. - Docs: Customize base job templates, Docker worker, Kubernetes worker - If you’re using flow.serve() (local, no work pool) - Set OS environment variables in the shell before starting your process;

serve()

inherits the parent process env. - Example:

export MLFLOW_TRACKING_URI="<http://mlflow.local:5000>"
    python flows/serve_ml.py

- Docs: Run flows in local processes - For shared non-sensitive config across flows - Use Prefect Variables (not environment variables, but convenient for config you fetch at runtime). - Example:

# set once
    prefect variable set mlflow_tracking_url "<http://mlflow.prod:5000>"

    # in your flow
    from prefect.variables import Variable
    url = Variable.get("mlflow_tracking_url")

- Docs/CLI: Variables - For secrets (API keys, tokens) - Do not put secrets in

job_variables

or YAML. Use a Secret block (or your secret manager) and load it at runtime. - Docs: Blocks Quick rule of thumb: - Per-deployment config: deployment

job_variables.env

- Infra-wide defaults: work pool base job template env - Local serve(): set OS env before starting - Shared config: Prefect Variables - Sensitive values: Secret blocks If you can share how you’re running flows (serve vs deploy) and which worker type (Docker/Kubernetes/Process), I can give you the exact snippet to use.

Awesome — you’ve got the hard parts running. Here’s the end-to-end path to deploy your flows from a Docker image to your Kubernetes worker using your existing prefect.yaml and

prefect deploy --all

. High level: - Build and push a Docker image that contains your flows - Point your deployments to that image via

work_pool.job_variables.image

- Deploy with

prefect deploy --all

- Trigger runs; your Kubernetes worker will launch Jobs using that image Step-by-step 1) Confirm your Kubernetes work pool - In the UI: check the work pool your worker is polling - Or via CLI:

prefect work-pool ls

Note the pool name (e.g., “kubernetes” or “my-k8s-pool”). Your deployments must target this pool. 2) Pick one of two image strategies - A. Prefect builds and pushes the image during deploy (recommended while migrating) - Add build/push steps to prefect.yaml (example below) - B. Your CI builds and pushes; prefect.yaml just references the prebuilt image - Skip build/push; point deployments at your image tag 3) Update prefect.yaml Option A: Build and push image during deploy

prefect-version: 3.0.0
name: my-flows

# Optional: produce a nice tag (e.g., short git sha)
build:
  - prefect.deployments.steps.run_shell_script:
      id: git-sha
      script: git rev-parse --short HEAD
      stream_output: false

  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.4.0
      image_name: "{{ $PREFECT_IMAGE_NAME }}"     # set this env var before deploy
      tag: "{{ git-sha.stdout }}"                 # or "latest" or a timestamp
      dockerfile: auto                            # auto-Dockerfile from your repo
      platform: "linux/amd64"

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.4.0
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"

# Runtime step so the flow process starts in the right directory inside the image
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/flows

# Optional shared def: point all deployments to the same k8s pool + image
definitions:
  work_pool: &k8s_pool
    name: "my-k8s-pool"                 # use your actual pool name
    job_variables:
      image: "{{ build-image.image }}"  # resolve to the built image
      namespace: "prefect"              # target namespace for Jobs
      image_pull_policy: "IfNotPresent"
      env:                              # put your flow env vars here
        MLFLOW_TRACKING_URI: "<http://mlflow.prod:5000>"

deployments:
  - name: train-prod
    entrypoint: flows/train.py:train
    work_pool: *k8s_pool
    parameters:
      epochs: 5

  - name: evaluate-prod
    entrypoint: flows/evaluate.py:evaluate
    work_pool: *k8s_pool

Option B: Use a prebuilt image - Skip build/push sections and set the image directly:

definitions:
  work_pool: &k8s_pool
    name: "my-k8s-pool"
    job_variables:
      image: "<http://1234567890.dkr.ecr.us-east-1.amazonaws.com/my-flows:2026-04-08|1234567890.dkr.ecr.us-east-1.amazonaws.com/my-flows:2026-04-08>"
      namespace: "prefect"
      env:
        MLFLOW_TRACKING_URI: "<http://mlflow.prod:5000>"

deployments:
  - name: train-prod
    entrypoint: flows/train.py:train
    work_pool: *k8s_pool

Notes: - If your registry is private, make sure your cluster can pull the image: - Attach an imagePullSecret to the Job via the work pool base job template, or - Use a service account with the right pull secret in the target namespace - To set more env vars, add them under

work_pool.job_variables.env

(deployment-level can override pool defaults) 4) Deploy your flows - If building/pushing in deploy, ensure Docker is running and you’re logged in to your registry - Set the image name when using Option A:

export PREFECT_IMAGE_NAME=<REGISTRY>/<REPO>     # e.g., <http://1234.dkr.ecr.us-east-1.amazonaws.com/my-flows|1234.dkr.ecr.us-east-1.amazonaws.com/my-flows>
  prefect deploy --all

- Watch output; you should see build → push → deployment creation

5) Trigger a test run

prefect deployment ls
prefect deployment run "<flow-name>/<deployment-name>"

- In the cluster, you should see a Job and Pod appear in the target namespace:

kubectl get jobs,pods -n prefect
  kubectl logs -n prefect job/<job-name>

6) Keep using the same workflow on code updates - Commit code changes - Re-run:

export PREFECT_IMAGE_NAME=<REGISTRY>/<REPO>   # Option A only
  prefect deploy --all

- This will rebuild/push a new image (Option A) or just update deployments to point at your chosen tag (Option B) 7) Where to put environment variables (MLflow, etc.) - Put per-deployment runtime env vars under

work_pool.job_variables.env

in prefect.yaml (as shown above) - Pool-wide defaults can be set in the work pool’s base job template; deployment-level

env

overrides them - For secrets, use a Secret block or Kubernetes Secret rather than plain env in YAML Helpful docs - Deployments: Deployments - Kubernetes worker: Kubernetes worker guide - prefect.yaml & deployment steps: Projects and prefect.yaml - Job variables: Customize job variables - Helm chart (worker): prefect-helm If you share: - Your work pool name - Whether you’ll build during deploy (Option A) or use a CI-built image (Option B) - Your target namespace and image registry …I can tailor the exact prefect.yaml for your setup (including imagePullSecrets and service account details if needed).

Short answer: use “deployments as code” with prefect.yaml + CI, and manage the run-form fields via the work pool’s base job template. This gives you versioned, repeatable updates to both code and the variables users see in the UI, while still allowing ad‑hoc overrides at run time. Recommended pattern for your use case - Source of truth in Git: - Keep your deployments in prefect.yaml (parameters and job_variables). - Keep a base-job-template.json for your Kubernetes work pool to define the fields that appear in the UI (variables with titles, descriptions, defaults). Commit both to your repo. - CI/CD on every change: - Build and push a Docker image with an immutable tag (e.g., git SHA). - Run prefect deploy --all to update deployments to the new image and defaults (parameters + job_variables). - Runtime overrides: - Users can still tweak parameters or job variables per run in the UI “Custom run” form. - When to use which knob: - Per-deployment defaults you want reviewed/versioned → prefect.yaml (requires redeploy to persist) - Shared infra defaults that change often across many deployments → set in the work pool’s base job template; updating the pool applies to subsequent runs without redeploy - One-off changes for a single run → override in the run form What to put where - Application inputs that belong in the form → deployment parameters (in prefect.yaml). They show as fields in the run form automatically. - Infra-level settings (image, namespace, resources, env) you want in the form → define them as variables in the work pool’s base job template (with titles/descriptions). They’ll render as structured fields instead of a raw JSON box. - Per-deployment infra defaults (e.g., specific MLflow URL per deployment) → job_variables in prefect.yaml; they’ll become the run-form defaults for that deployment and can be overridden per run. - Shared values across many deployments that you want to change without redeploy → set as defaults in the work pool’s base job template or pool variables. - Secrets → Kubernetes Secrets or Prefect Secret blocks; reference them from job_variables (via valueFrom or block refs), not plain text. Minimal example layout - base-job-template.json (checked in; controls which fields appear in the form) - Update the pool when this changes:

prefect work-pool create my-k8s-pool --type kubernetes --base-job-template base-job-template.json --overwrite

- prefect.yaml (deployment & defaults)

prefect-version: 3.0.0
  name: my-flows

  build:
    - prefect_docker.deployments.steps.build_docker_image:
        id: build-image
        requires: prefect-docker>=0.4.0
        image_name: "{{ $PREFECT_IMAGE_NAME }}"
        tag: "{{ $GIT_SHA }}"
        dockerfile: auto

  push:
    - prefect_docker.deployments.steps.push_docker_image:
        requires: prefect-docker>=0.4.0
        image_name: "{{ build-image.image_name }}"
        tag: "{{ build-image.tag }}"

  pull:
    - prefect.deployments.steps.set_working_directory:
        directory: /opt/prefect/flows

  definitions:
    work_pool: &k8s_pool
      name: "my-k8s-pool"
      job_variables:
        image: "{{ build-image.image }}"
        namespace: "prefect"
        image_pull_policy: "IfNotPresent"
        env:
          MLFLOW_TRACKING_URI: "<http://mlflow.prod:5000>"

  deployments:
    - name: train-prod
      entrypoint: flows/train.py:train
      work_pool: *k8s_pool
      parameters:
        epochs: 5

Operational loop - Change flow code or deployment defaults → commit - If you changed the form fields (pool variables), update the work pool from your base-job-template.json - Build and push image (CI), then:

export PREFECT_IMAGE_NAME=<registry/repo>
  export GIT_SHA=$(git rev-parse --short HEAD)
  prefect deploy --all

- New runs use the updated image and updated defaults; users can still tweak values per run in the UI. Why this fits your needs

- Diffable, reviewable changes to both code and defaults - Fields in the run form are stable and intentional (driven by the pool template) - Frequent changes are easy: - Deployment-specific → edit prefect.yaml + redeploy - Global across many deployments → edit pool defaults (no redeploy needed) - One-off run → override in UI Helpful docs - Deployments: Deployments - Projects and prefect.yaml: Projects and prefect.yaml - Customize job variables: Customize job variables - Customize base job templates: Customize base job templates - Kubernetes worker: Kubernetes worker guide If you share a sample of the variables you want in the form, I can sketch a base-job-template.json and prefect.yaml tailored to your deployments.

Got it — if you’re already defining all app inputs as Pydantic models, you don’t need to duplicate them in a work pool template. Recommended approach - Keep application config as Pydantic parameters - These auto-populate the “Parameters” section in the run form. - Update defaults by changing code and running your usual prefect deploy --all. - For libraries that expect environment variables (e.g., MLflow) - Pass them as parameters and set the env early in the flow. This keeps a single source of truth (your Pydantic model) and still satisfies libs that read from os.environ. Example

from pydantic import BaseModel
from prefect import flow

class TrainConfig(BaseModel):
    mlflow_tracking_uri: str
    experiment_name: str = "default"

@flow
def train(cfg: TrainConfig):
    import os
    os.environ["MLFLOW_TRACKING_URI"] = cfg.mlflow_tracking_uri
    os.environ["MLFLOW_EXPERIMENT_NAME"] = cfg.experiment_name
    # ...rest of your flow...

- Keep infrastructure knobs as job variables, without a custom template - Image/namespace/resources belong in job variables; you don’t have to maintain a base-job-template.json to use them. - Put sensible defaults in prefect.yaml so they show up as defaults in the UI’s “Job variables” JSON editor and can be overridden per run. prefect.yaml snippet

definitions:
  work_pool: &k8s_pool
    name: "my-k8s-pool"
    job_variables:
      image: "<http://1234567890.dkr.ecr.us-east-1.amazonaws.com/my-flows:{{|1234567890.dkr.ecr.us-east-1.amazonaws.com/my-flows:{{> $GIT_SHA }}"
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"
      env:
        # Optional: only if you still want a container-level default
        MLFLOW_TRACKING_URI: "<http://mlflow.prod:5000>"

deployments:
  - name: train-prod
    entrypoint: flows/train.py:train
    work_pool: *k8s_pool
    # Parameter defaults stay in code (Pydantic), not here

Why this fits your constraint - No duplication: parameters live only in code; they auto-render in the UI. - You still control infra via job variables (no template file required), and can override per run in the UI’s JSON box. - Updates are the same loop you already use: change code → prefect deploy --all. If you change job variable defaults, edit prefect.yaml and redeploy. Optional convenience - If you want pool-wide defaults (e.g., shared imagePullPolicy or global env), set them once in the Work Pool UI; deployments can still override. This also avoids a template file. Docs - Deployments: Deployments - Kubernetes worker: Kubernetes worker guide - Customize job variables (if/when you want structured fields): Customize job variables If you share your current Pydantic model (fields/names), I can show the minimal prefect.yaml and a pattern to set just the necessary env vars from those parameters.

Great setup. Since your GitHub Action already builds and pushes the image, you don’t need the build/push sections in prefect.yaml at all. Here’s how they work and how to adapt your flow for your CI-built images. What build/push do in prefect.yaml - Build steps run locally when you execute prefect deploy. They build a Docker image (often using a generated “auto” Dockerfile) and return metadata like image_name, tag, image. - Push steps run after build during the same prefect deploy and push that image to your registry. - You can then template those outputs into your deployments (e.g., job_variables.image: "{{ build-image.image }}") so deployments point to the fresh image. Since your CI already builds/pushes: - Remove build and push from prefect.yaml and just point deployments to the prebuilt image tag. - You still need to run prefect deploy --all to update the Prefect deployment to the new image tag so future runs use your updated code. Two clean ways to wire this with your CI 1) Pass the image tag into prefect.yaml via an environment variable - prefect.yaml:

prefect-version: 3.0.0
  name: my-flows

  # Optional at-runtime convenience step:
  pull:
    - prefect.deployments.steps.set_working_directory:
        directory: /opt/prefect/flows

  definitions:
    work_pool: &k8s_pool
      name: "my-k8s-pool"
      job_variables:
        image: "{{ $IMAGE }}"       # CI injects this
        namespace: "prefect"
        image_pull_policy: "IfNotPresent"

  deployments:
    - name: train
      entrypoint: flows/train.py:train
      work_pool: *k8s_pool
    - name: evaluate
      entrypoint: flows/evaluate.py:evaluate
      work_pool: *k8s_pool

- In your GitHub Action after pushing the image:

env:
    IMAGE: <http://ghcr.io/org/repo:${{|ghcr.io/org/repo:${{> github.sha }}   # or your release tag
  run: |
    prefect cloud login --key "$PREFECT_API_KEY" --workspace "$PREFECT_WORKSPACE"
    prefect deploy --all

2) Hardcode a template that reads the tag from a script or file - Keep no build/push steps, but add a tiny run_shell_script step to compute the tag (e.g., read a VERSION file your CI writes), then reference it:

build:
    - prefect.deployments.steps.run_shell_script:
        id: get-tag
        script: cat VERSION
        stream_output: false

  definitions:
    work_pool: &k8s_pool
      name: "my-k8s-pool"
      job_variables:
        image: "<http://ghcr.io/org/repo:{{|ghcr.io/org/repo:{{> get-tag.stdout }}"

- Your CI writes VERSION to the repo/workspace before calling prefect deploy. How this deploys code changes - Your flow code lives in the image. Updating code = build/push a new image tag, then update deployments to reference that new tag. - prefect deploy --all reads prefect.yaml and updates the deployment’s job_variables.image to the tag you provided (via env or script). - The Kubernetes worker will then launch each new flow run using that image. If you reuse a mutable tag (e.g., latest), set image_pull_policy: "Always" and understand the risks; best practice is immutable tags (e.g., git SHA). Do you still need pull steps? - Not required for images that already contain your code, but the common set_working_directory step ensures the process starts in the directory where your code lives in the image. It’s a harmless convenience. Pitfalls to avoid - Not redeploying: If you use immutable tags and don’t run prefect deploy, your deployments still point to the old tag and won’t pick up new code. - Using latest with IfNotPresent: The node’s cache may mask updates. If you must use latest, set image_pull_policy: "Always". - Private registry pulls: Ensure imagePullSecrets or node auth is configured so the worker’s Jobs can pull the image. Docs for reference - Deployments: Deployments - Projects and prefect.yaml: Projects and prefect.yaml

- Kubernetes worker: Kubernetes worker guide If you share your image naming convention (e.g., ghcr.io/org/repo:${{ github.sha }} or release tag) I can drop in the exact prefect.yaml and a minimal GitHub Action snippet for your flow.

Great question. Here’s a clean, version-controlled workflow that uses build/push steps in prefect.yaml and keeps deployments in lockstep with your code. Recommended workflow - Build on every push to main with immutable tags - Let prefect.yaml’s build/push steps create/push an image tagged with the commit SHA. - Then run prefect deploy --all to update deployments to that image. Each deploy is tied to a specific commit. - Promote on release tags without rebuilding - Keep your existing release flow: when you create a git tag, retag the already-built SHA image in your registry with the semver (e.g., v1.2.3) and optionally run a promotion deploy (point prod deployments to that semver tag). - This gives you both immutable SHAs for traceability and human-friendly semver aliases. Why this is nice - One source of truth: prefect.yaml (code + infra defaults). - Immutable images per commit; easy rollbacks and diffs. - Releases are just tags/retags; no rebuild needed for prod promotion. prefect.yaml (build/push + K8s deployments) - Parameters stay in code (your Pydantic models). - Image/tag comes from the repo and commit.

prefect-version: 3.0.0
name: my-flows

# Produce a short git sha for tagging
build:
  - prefect.deployments.steps.run_shell_script:
      id: git-sha
      script: git rev-parse --short HEAD
      stream_output: false

  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.4.0
      image_name: "{{ $PREFECT_IMAGE_NAME }}"   # provided by CI
      tag: "{{ git-sha.stdout }}"               # immutable per commit
      dockerfile: auto
      platform: "linux/amd64"

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.4.0
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"

# Convenience: ensure the flow process starts in your code dir in the image
pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/flows

definitions:
  work_pool: &k8s_pool
    name: "my-k8s-pool"               # your pool name
    job_variables:
      image: "{{ build-image.image }}" # points deployments at the new image
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"

deployments:
  - name: train
    entrypoint: flows/train.py:train
    work_pool: *k8s_pool

  - name: evaluate
    entrypoint: flows/evaluate.py:evaluate
    work_pool: *k8s_pool

GitHub Actions (CI) examples 1) On push to main: build, push, deploy to Prefect - Builds a SHA-tagged image and updates deployments to that SHA.

name: Deploy (main)

on:
  push:
    branches: [ main ]

jobs:
  deploy-main:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    env:
      PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
      PREFECT_WORKSPACE: ${{ secrets.PREFECT_WORKSPACE }} # e.g. acct/workspace
      PREFECT_IMAGE_NAME: <http://ghcr.io/your-org/your-repo|ghcr.io/your-org/your-repo>      # registry/repo
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install Prefect + Docker integration
        run: |
          pip install "prefect>=3" "prefect-docker>=0.4.0"

      - name: Auth to GHCR (or your registry)
        run: |
          echo "${{ secrets.GITHUB_TOKEN }}" | docker login <http://ghcr.io|ghcr.io> -u ${{ github.actor }} --password-stdin

      - name: Login to Prefect Cloud
        run: prefect cloud login --key "$PREFECT_API_KEY" --workspace "$PREFECT_WORKSPACE"

      - name: Deploy all
        env:
          PREFECT_IMAGE_NAME: ${{ env.PREFECT_IMAGE_NAME }}
        run: prefect deploy --all

2) On release tag: retag and optionally promote - No rebuild; just retag the SHA image (from the main build) and, if desired, update deployments to use the semver tag for prod. Option A: Just retag in the registry ``` name: Retag (release) on: push: tags: - 'v*.*.*' jobs: retag: runs-on: ubuntu-latest

steps: - name: Extract commit SHA of tag run: echo "SHA=${GITHUB_SHA::7}" >> $GITHUB_ENV - name: Login to GHCR run: | echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin - name: Pull, retag, push run: | docker pull ghcr.io/your-org/your-repo:${SHA} docker tag ghcr.io/your-org/your-repo:${SHA} ghcr.io/your-org/your-repo:${{ github.ref_name }} docker push ghcr.io/your-org/your-repo:${{ github.ref_name }}

Option B: Promote deployments to the release tag
- Keep a second prefect.yaml variant or pass IMAGE via env to point prod to the release tag (no build/push sections, just overrides).
- Minimal promotion deploy (same repo, same deployments, different image reference):

name: Promote (release) on: push: tags: - 'v*.*.*' jobs: promote: runs-on: ubuntu-latest env: PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }} PREFECT_WORKSPACE: ${{ secrets.PREFECT_WORKSPACE }} IMAGE: ghcr.io/your-org/your-repo:${{ github.ref_name }} steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: '3.11' - run: pip install "prefect>=3" - run: prefect cloud login --key "$PREFECT_API_KEY" --workspace "$PREFECT_WORKSPACE" - name: Override image to release tag and deploy run: | # Simple sed-based override; or maintain a small prod-prefect.yaml sed -i 's|image: ".*"|image: "'"$IMAGE"'"|g' prefect.yaml prefect deploy --all ``` Notes and best practices - Immutable tags: prefer git SHA for deployments. If you must use a mutable tag (like latest), set image_pull_policy: "Always". - Keep parameters in code: your Pydantic models auto-populate the UI; no duplication needed. - Registry pulls: ensure Kubernetes can pull from your registry (imagePullSecrets or node auth). - Multi-env: point different deployments or work pools at different images (e.g., staging uses SHA, prod uses semver), or run the “promote” job to switch prod to the release tag. Docs for reference - Deployments: Deployments - Projects and prefect.yaml: Projects and prefect.yaml - Kubernetes worker: Kubernetes worker guide - Customize job variables: Customize job variables If you share your registry and current Action, I can tailor the exact prefect.yaml and Action steps (including imagePullSecrets or separate staging/prod pools).

Short answer - Yes: you can have your GitHub Action run prefect deploy --all and let the Prefect Docker step build/push the image as part of that deploy. - You don’t have to give up buildx; Prefect’s build step supports both docker-py and buildx (with SSH + secrets), so private repos/registries still work. - If you already have a custom Dockerfile that pulls packages from private repos, keep using it; Prefect just orchestrates docker build/push for you. How it fits together - Your Action is the driver. It: 1) logs into your registry 2) installs Prefect + prefect-docker (and python-on-whales if you want buildx) 3) runs prefect deploy --all - prefect deploy reads prefect.yaml, runs the build/push steps (using your Dockerfile or “auto”), and updates deployments to the new image tag. prefect.yaml (using your own Dockerfile + buildx, with support for secrets/ssh)

prefect-version: 3.0.0
name: my-flows

build:
  - prefect.deployments.steps.run_shell_script:
      id: git-sha
      script: git rev-parse --short HEAD
      stream_output: false

  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.4.0
      image_name: "{{ $PREFECT_IMAGE_NAME }}"     # set in CI env
      tag: "{{ git-sha.stdout }}"
      dockerfile: Dockerfile                      # your Dockerfile (not “auto”)
      build_backend: buildx                       # enables BuildKit features
      buildargs:
        SOME_ARG: "value"                         # if needed
      ssh: default                                # forwards SSH agent for private git
      secrets:                                    # pass build-time secrets for private registries
        - id=pip_token,env=PIP_TOKEN
        - id=npm_token,env=NPM_TOKEN
      # optional caching
      # cache_from: ["type=gha"]
      # cache_to:   ["type=gha,mode=max"]

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.4.0
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/flows

definitions:
  work_pool: &k8s_pool
    name: "my-k8s-pool"
    job_variables:
      image: "{{ build-image.image }}"
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"

deployments:
  - name: train
    entrypoint: flows/train.py:train
    work_pool: *k8s_pool

GitHub Actions sketch

name: Deploy Prefect

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    env:
      PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
      PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}   # or use `prefect cloud login --workspace ...`
      PREFECT_IMAGE_NAME: <http://ghcr.io/your-org/your-repo|ghcr.io/your-org/your-repo>
      PIP_TOKEN: ${{ secrets.PIP_TOKEN }}               # for private PyPI
      NPM_TOKEN: ${{ secrets.NPM_TOKEN }}               # for private NPM (if applicable)
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - uses: docker/login-action@v3
        with:
          registry: <http://ghcr.io|ghcr.io>
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install Prefect + Docker integration
        run: |
          pip install "prefect>=3" "prefect-docker>=0.4.0" "python-on-whales>=0.81"

      - name: Deploy all
        run: prefect deploy --all

Handling private dependencies from other repos - Keep your custom Dockerfile. Prefect will call docker/buildx with your Dockerfile; you’re not limited to “auto”. - For private git repos: use buildx + SSH agent forwarding and Dockerfile mounts: - In prefect.yaml:

ssh: default

- In Dockerfile:

# syntax=docker/dockerfile:1.4
    FROM python:3.12-slim
    RUN --mount=type=ssh git clone git@github.com:your-org/private-repo.git /src

- For private package registries (pip/npm): use buildx secrets and read them during RUN: - In prefect.yaml:

secrets:
      - id=pip_token,env=PIP_TOKEN
      - id=npm_token,env=NPM_TOKEN

- In Dockerfile:

RUN --mount=type=secret,id=pip_token \
        PIP_INDEX_URL="<https://token:$(cat> /run/secrets/pip_token)@python.pkg.github.com/simple" \
        pip install -r requirements.txt

    RUN --mount=type=secret,id=npm_token \
        echo "//npm.pkg.github.com/:_authToken=$(cat /run/secrets/npm_token)" > ~/.npmrc && \
        npm ci

- If your base image is in a private registry, just docker login in the Action before prefect deploy. Alternative: keep your existing buildx job - If you prefer your current buildx step, keep it. After pushing the image, run prefect deploy --all and pass the tag via env: - In prefect.yaml:

image: "{{ $IMAGE }}"

- In Actions:

env:
      IMAGE: <http://ghcr.io/your-org/your-repo:$|ghcr.io/your-org/your-repo:$>{{ github.sha }}
    run: prefect deploy --all

Either path is valid. Using the Prefect Docker step centralizes the build/push logic in prefect.yaml, while keeping your custom Dockerfile and all the buildx features you need. Docs - Deployments: Deployments - Projects and prefect.yaml: Projects and prefect.yaml - Kubernetes worker: Kubernetes worker guide - Customize job variables: Customize job variables If you share a snippet of your current Dockerfile (sanitized), I can show the exact secrets/ssh lines to drop in and the matching prefect.yaml build config.

You’ve got three good ways to handle those custom wheels when moving the build into GitHub Actions + Prefect’s Docker step. Pick the one that fits your current wheel source and security model. Option A (keep your COPY *.whl pattern): pre-download wheels before the Docker build - Fetch the wheels into the repo workspace first, then let your existing Dockerfile COPY them as-is. - Do it either: - in GitHub Actions before prefect deploy, or - as a run_shell_script step before the Prefect build_docker_image step. Example: download before the build step via prefect.yaml

prefect-version: 3.0.0
name: my-flows

build:
  - prefect.deployments.steps.run_shell_script:
      id: fetch-wheels
      script: |
        mkdir -p wheels
        # Example: GitHub Releases (private) using a PAT
        curl -L -H "Authorization: token $GH_PAT" \
          -o wheels/custom_pkg-1.2.3-py3-none-any.whl \
          <https://github.com/your-org/your-repo/releases/download/v1.2.3/custom_pkg-1.2.3-py3-none-any.whl>
      stream_output: false

  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.4.0
      image_name: "{{ $PREFECT_IMAGE_NAME }}"
      tag: "{{ $GIT_SHA }}"
      dockerfile: Dockerfile
      build_backend: buildx
      ssh: default
      # optional: secrets, cache_from/to, buildargs, etc.

push:
  - prefect_docker.deployments.steps.push_docker_image:
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"

GitHub Actions step (set GH_PAT, PREFECT_IMAGE_NAME, GIT_SHA in env):

- name: Deploy with Prefect
  env:
    PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
    PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
    PREFECT_IMAGE_NAME: <http://ghcr.io/your-org/your-repo|ghcr.io/your-org/your-repo>
    GIT_SHA: ${{ github.sha }}
    GH_PAT: ${{ secrets.GH_PAT }}
  run: prefect deploy --all

Your Dockerfile can stay the same:

# syntax=docker/dockerfile:1.4
FROM prefecthq/prefect:3-python3.11
WORKDIR /opt/prefect/flows

# Wheels are now in the build context (downloaded by the step above)
COPY wheels/*.whl /wheels/
RUN pip install --no-cache-dir /wheels/*.whl

COPY . .

Option B (install wheels during docker build): use buildx secrets/SSH - Download or install in the Dockerfile using BuildKit mounts (great for private sources). - Pass tokens as build secrets; don’t bake them into the image. prefect.yaml (enable buildx + secrets):

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      image_name: "{{ $PREFECT_IMAGE_NAME }}"
      tag: "{{ $GIT_SHA }}"
      dockerfile: Dockerfile
      build_backend: buildx
      secrets:
        - id=gh_pat,env=GH_PAT        # from CI env
      ssh: default

Dockerfile example (GitHub Releases private):

# syntax=docker/dockerfile:1.4
FROM prefecthq/prefect:3-python3.11
WORKDIR /opt/prefect/flows

# Use secret to download; token never persists in the image
RUN --mount=type=secret,id=gh_pat \
    GH_PAT="$(cat /run/secrets/gh_pat)" && \
    curl -L -H "Authorization: token ${GH_PAT}" \
      -o /tmp/custom_pkg-1.2.3-py3-none-any.whl \
      <https://github.com/your-org/your-repo/releases/download/v1.2.3/custom_pkg-1.2.3-py3-none-any.whl> && \
    pip install --no-cache-dir /tmp/custom_pkg-1.2.3-py3-none-any.whl

COPY . .

Option C (skip local wheel files): pip install from your private index - Preferable if those wheels are also hosted on a private PyPI (or GH Packages PyPI). - Use buildx secrets to inject the token, and install directly by name/version. Dockerfile example (private PyPI):

# syntax=docker/dockerfile:1.4
FROM prefecthq/prefect:3-python3.11
WORKDIR /opt/prefect/flows

RUN --mount=type=secret,id=pip_token \
    PIP_INDEX_URL="<https://__token_>_:$(cat /run/secrets/pip_token)@pypi.mycompany.com/simple" \
    PIP_EXTRA_INDEX_URL="<https://pypi.org/simple>" \
    pip install --no-cache-dir custom-pkg==1.2.3 another-custom==4.5.6

COPY . .

In prefect.yaml: ``` build:

- prefect_docker.deployments.steps.build_docker_image: id: build-image image_name: "{{ $PREFECT_IMAGE_NAME }}" tag: "{{ $GIT_SHA }}" dockerfile: Dockerfile build_backend: buildx secrets: - id=pip_token,env=PIP_TOKEN ``` Recommendations - If you want to keep your current Dockerfile unchanged, Option A is the least invasive: add a pre-build download step (either in GitHub Actions or in prefect.yaml via run_shell_script). - If you’re comfortable editing the Dockerfile, Option B/C are more reproducible and keep secrets out of the build context using BuildKit mounts. - Use immutable tags (e.g., git SHA) and point deployments to that tag in prefect.yaml. Then run prefect deploy --all from GitHub Actions to update deployments. - If you ever switch to a mutable tag (e.g., latest), set image_pull_policy: Always in job_variables. Helpful docs - Deployments: Deployments - Projects and prefect.yaml: Projects and prefect.yaml - Kubernetes worker: Kubernetes worker guide - CI/CD guide: Deploy via CI/CD - Customize job variables: Customize job variables If you share where those wheels are hosted (GitHub Releases, S3, private PyPI, etc.), I can give you the exact curl/pip/secret config to drop into either the pre-build step or Dockerfile.

Two easy ways to use the Git tag (e.g., 1.1.0) as your image tag. Option A: Prefect builds/pushes the image and uses the Git tag - Pass the tag to prefect.yaml via an env var from your GitHub Action, then use it in the build step. prefect.yaml

prefect-version: 3.0.0
name: my-flows

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build-image
      requires: prefect-docker>=0.4.0
      image_name: "{{ $PREFECT_IMAGE_NAME }}"
      tag: "{{ $IMAGE_TAG }}"            # <- use the git tag here
      dockerfile: Dockerfile
      # optional: also tag by SHA for traceability
      additional_tags:
        - "{{ $GITHUB_SHA }}"

push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.4.0
      image_name: "{{ build-image.image_name }}"
      tag: "{{ build-image.tag }}"
      additional_tags: "{{ build-image.additional_tags }}"

definitions:
  work_pool: &k8s_pool
    name: "my-k8s-pool"
    job_variables:
      image: "{{ build-image.image }}"
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"

deployments:
  - name: train
    entrypoint: flows/train.py:train
    work_pool: *k8s_pool

GitHub Actions (tag push)

on:
  push:
    tags:
      - '*'

jobs:
  deploy:
    runs-on: ubuntu-latest
    env:
      PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
      PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
      PREFECT_IMAGE_NAME: <http://ghcr.io/your-org/your-repo|ghcr.io/your-org/your-repo>
      # Use the tag name; strip leading "v" if you tag releases like v1.1.0
      IMAGE_TAG: ${{ github.ref_name }}
      GITHUB_SHA: ${{ github.sha }}
    steps:
      - uses: actions/checkout@v4

      - uses: docker/login-action@v3
        with:
          registry: <http://ghcr.io|ghcr.io>
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install "prefect>=3" "prefect-docker>=0.4.0"

      - name: Deploy all
        run: prefect deploy --all

Notes: - If your tags are like v1.1.0 and you want 1.1.0, set IMAGE_TAG: ${{ github.ref_name && startsWith(github.ref_name, 'v') && replace(github.ref_name, 'v', '') || github.ref_name }} or do a small shell step to trim “v”. - additional_tags lets you keep both the semver and the SHA pointing at the same image for traceability. Option B: Keep your existing build/push and only update deployments - If you’d rather still build in a separate job (or with buildx directly), just pass the prebuilt image reference to prefect.yaml. prefect.yaml

prefect-version: 3.0.0
name: my-flows

definitions:
  work_pool: &k8s_pool
    name: "my-k8s-pool"
    job_variables:
      image: "{{ $IMAGE }}"        # e.g., <http://ghcr.io/org/repo:1.1.0|ghcr.io/org/repo:1.1.0>
      namespace: "prefect"
      image_pull_policy: "IfNotPresent"

deployments:
  - name: train
    entrypoint: flows/train.py:train
    work_pool: *k8s_pool

GitHub Actions (after your own build/push finishes)

env:
  IMAGE: <http://ghcr.io/your-org/your-repo:${{|ghcr.io/your-org/your-repo:${{> github.ref_name }}
run: |
  pip install "prefect>=3"
  prefect deploy --all

Best practices - Prefer immutable tags (the git tag is great). If you ever use a mutable tag (e.g., latest), set image_pull_policy: Always. - You can keep both: tag the image with the git tag and the SHA (via additional_tags) to make audits and rollbacks easy. - If you fetch custom wheels or private deps during the build, keep your Dockerfile as-is and let Prefect’s build step call docker with your Dockerfile; or use a pre-build script (run_shell_script) to download assets before the build. Related docs - Deployments - Projects and prefect.yaml

- Deploy via CI/CD If you share your current tagging scheme (e.g., v1.1.0 vs 1.1.0) and whether you want to keep a SHA tag too, I can tailor the exact YAML and Action snippet.