<@ULVA73B9P> Hi hope you're all doing well. I'm re...
# ask-marvin
j
@Marvin Hi hope you're all doing well. I'm reaching out because I could really use some help. I previously had a self-hosted Prefect instance running version 2.14.21. We were told to migrate to Prefect 3, so all my existing flows needed to be moved over — the version installed at the time was 3.6.17. On the code side, I've only had to make a couple of changes, but there's one error I've been stuck on for weeks now. My flows on version 2 use pre-built Docker images (since my self-hosted instance runs on Kubernetes), and they also have some additional dependencies — things like ChromeDriver installation, special configurations, etc. However, I can't get them to work. The errors I get say either that the dependencies don't exist, or the flow just fails outright. I tried spinning up a flow from scratch and everything worked fine, but as soon as I reference any of the dependencies that are supposed to be installed in the image, it falls back into the same "dependencies not found" error. One thing I tried was setting up the flow using a prefect.yaml where I do a
git_clone
of the repository, but I keep hitting the same issue. I always provide the correct Docker image along with its tag, and I've even run the image locally on my machine and it works fine.
m
thought for 276.4 seconds
Thanks for the detailed context — this is a common sticking point when moving from 2.x infra blocks to 3.x work pools. From what you’ve described, there are two very likely culprits: 1) Your flow runs are not actually using your custom image (falling back to the work pool’s base job template image — usually a Prefect image), so your ChromeDriver/extra deps aren’t present. 2) A build/push step or default behavior during deployment is unintentionally replacing/overriding the image you intended to use. Below is a targeted checklist and working examples to help you verify and fix this. Quick verification checklist - Confirm you are using a Kubernetes work pool and a running Kubernetes worker for that pool. - Confirm the deployment is configured to use your image: - Run:
Copy code
prefect deployment inspect "<flow-name>/<deployment-name>"
Look for job variables and confirm the image value is exactly your image:tag. - Run:
Copy code
prefect work-pool inspect "<pool-name>" --output json
Confirm the pool’s base_job_template.default image isn’t what your runs are using unintentionally. - Check the actual image used by a failing pod: - Find the pod for a failed flow run, then:
Copy code
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].image}{"\n"}'
If this isn’t your image, the deployment isn’t picking up your override. - Check worker logs (the Kubernetes worker pod) right before it submits the job — it will log the job spec/image it’s about to launch. The most common fixes A) Use a prebuilt image without building anything during deploy - If you already have a vetted prebuilt image (with ChromeDriver etc.), do not run build/push steps and do not let deploy rebuild an image for you. - In Prefect 3.x, Flow.deploy defaults to build=True/push=True. If you deploy via Python, set build=False/push=False or remove build/push steps from prefect.yaml. Minimal prefect.yaml (prebuilt image, no build/push)
Copy code
prefect-version: "3.0"

# No build:, no push:

# Optional: pull your code at runtime (ensure git is in your image if you use git_clone)
pull:
  - prefect.deployments.steps.git_clone:
      repository: "<https://github.com/your-org/your-repo.git>"
      branch: "main"

deployments:
  - name: my-k8s-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: your-k8s-pool
      job_variables:
        image: "your-registry/your-image:your-tag"
        namespace: "production"
        # add anything else you need here, e.g. service_account_name, resources, env, image_pull_secrets
Then:
Copy code
prefect deploy
B) Deploy via Python with a prebuilt image
Copy code
from prefect import flow

@flow
def my_flow():
    # your logic
    pass

if __name__ == "__main__":
    my_flow.from_source(
        source="<https://github.com/your-org/your-repo.git>",
        entrypoint="flows/my_flow.py:my_flow",
    ).deploy(
        name="my-k8s-deployment",
        work_pool_name="your-k8s-pool",
        image="your-registry/your-image:your-tag",
        build=False,   # IMPORTANT: don't rebuild your image
        push=False,    # IMPORTANT: don't push a new image
        job_variables={
            "namespace": "production",
        },
    )
C) Deploy from CLI and override image explicitly - Verified CLI command:
Copy code
prefect deploy [./path/to/file.py:flow_func] \
  --pool your-k8s-pool \
  --job-variable image=your-registry/your-image:your-tag
- CLI flags override prefect.yaml job_variables. This is a great sanity check to ensure the image is set at deployment time. D) If you do want Prefect to build an image - Make sure your build step actually installs your system dependencies (Chrome, ChromeDriver, etc.). Many folks accidentally build a “vanilla” Prefect image that lacks those extras — then wonder why it fails at runtime.
- If you pass image=... but leave build=True, Prefect may still build and tag the image (depending on your config). If your goal is to use a known-good prebuilt, explicitly set build=False/push=False. E) If you use git_clone pull steps - Ensure git is present in your runtime image. The git_clone step runs in the ephemeral runtime environment before your flow executes. - If your image does not have git, either: - Add git to your image, or - Bake your code into the image and omit pull steps, or - Use another storage/pull method that doesn’t require git in the container. Helpful commands (verified) - Show available work pools:
Copy code
prefect work-pool ls
- Inspect a work pool (to check its base job template image and settings):
Copy code
prefect work-pool inspect "your-k8s-pool" --output json
- Inspect a deployment:
Copy code
prefect deployment inspect "your-flow-name/your-deployment-name"
A few more gotchas I see in migrations - Using the wrong pool type: If you point the deployment at a Process pool (or any non-Kubernetes pool), it will not use your Kubernetes image. - Using the pool default image unintentionally: If you do not set job_variables.image at the deployment, the pool’s base job template image will be used instead. - Private registries: if your image is private, confirm imagePullSecrets are set in job_variables so the pod can pull. - Tags like “latest”: Kubernetes may use cached images. Prefer unique tags or set imagePullPolicy to Always in the job template if needed. If you can share: - Your prefect.yaml (just the deployments and any build/push/pull sections) - The output of: -
prefect deployment inspect "<flow>/<deployment>"
-
prefect work-pool inspect "<pool>" --output json
- One example pod’s actual image: -
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].image}{"\n"}'
…I can pinpoint exactly where the override is getting lost and give you a precise fix. Lastly, to call out the 2.x → 3.x change explicitly: - 2.x infra blocks (like KubernetesJob/DockerContainer with an image field) were replaced by 3.x work pools with a base job template + per-deployment job_variables. Don’t use the old 2.x `prefect deployment build`; in 3.x you should use
prefect deploy
,
flow.from_source(...).deploy(...)
, and set
work_pool.job_variables.image
(or pass
--job-variable image=...
).