<@ULVA73B9P> I'm working with kubernetes workers a...
# ask-marvin
s
@Marvin I'm working with kubernetes workers and need to dynamically set the docker image used to run deployments. I'm using GitOps processes which means that the version of the image to use is updated in my infrastructure git. What is the recommended way to inform the deployment to use an image based on that value ?
m
thought for 289.5 seconds
Great question. In Prefect 3.x with Kubernetes workers, the cleanest way to “point” flow runs at a Docker image that’s managed by your GitOps system is to drive the image via job variables, not hard-code it in the deployment. There are three common patterns—pick the one that fits how broadly you want the change to apply: 1) Single image for a whole work pool (recommended if all deployments in a pool share the same base image) - Put the image into the Kubernetes work pool’s default job variables and reference it in the pool’s base job template. - Example base job template snippet:
Copy code
# Work pool base job template (Kubernetes)
pod:
  spec:
    containers:
      - name: prefect
        image: "{{ variables.image | default('<http://ghcr.io/your-org/your-image:latest|ghcr.io/your-org/your-image:latest>') }}"
        imagePullPolicy: Always
- Then set the work pool’s default job variable
image
to your GitOps-managed tag (e.g., `ghcr.io/your-org/your-image:1.2.3`). - Your GitOps pipeline updates this single value via the Prefect API/UI whenever the tag changes. No worker restart is needed; new flow runs pick it up automatically. Pros: - Centralized control: change once, all deployments in the pool use the new image. - Minimal code changes. 2) Per-deployment image (if each deployment should track its own image) - Keep the template line above in your pool and set the deployment’s job variables with the image:
Copy code
flow = Flow.from_source("<https://github.com/your-org/your-repo>", entrypoint="flows/my_flow.py:my_flow")
flow.deploy(
  name="my-deployment",
  work_pool_name="k8s-pool",
  job_variables={"image": "<http://ghcr.io/your-org/your-image:1.2.3|ghcr.io/your-org/your-image:1.2.3>"},
  build=False,
  push=False,
)
- Your GitOps pipeline updates the deployment’s
job_variables.image
when the tag changes (via API/UI). This overrides the pool default for that deployment only. Pros: - Granular control per deployment. 3) Per-run override (for canaries or one-off runs) - When triggering a run, pass a job variable override like
{"image": "<http://ghcr.io/...:test-sha|ghcr.io/...:test-sha>"}
so just that run uses the new image. This is great for testing before rolling out broadly. Additional notes - Don’t bake the image tag into code if GitOps is your source of truth; leave builds off (
build=False
,
push=False
) and just point Prefect at your pre-built image. - Ensure
imagePullPolicy: Always
so new pods pull the updated tag on each run. - Precedence of values: per-run job variables > deployment job variables > work pool default job variables. - If all flows share the same image version, Pattern 1 (work pool default) is the simplest and most “GitOps-native”. If it helps, I can show: - A small Python snippet to update a work pool’s default
image
via the Prefect API, or - How to structure a GitOps job that reads a tag file in your infra repo and updates the pool/deployment accordingly. A couple questions to tailor the approach: - Do you want a single image version across all deployments in a pool, or different images per deployment? - Will your GitOps pipeline be able to call Prefect’s API to update work pool/deployment settings when the tag changes?