<@ULVA73B9P> Do the prefect-worker images pin the ...
# ask-marvin
d
@Marvin Do the prefect-worker images pin the prefect version?
m
thought for 256.9 seconds
Short answer: not by default. - There isn’t a separate “prefect-worker” image. Workers typically run in the general Prefect image:
prefecthq/prefect
. - Floating tags move forward as new releases come out: -
:latest
→ latest stable Prefect of the latest major -
:3-latest
→ latest Prefect 3.x -
:3-python3.12
(and similar) → latest Prefect 3.x for that Python minor - Pinned tags are versioned and do not change: -
:3.1.2
or
:3.1.2-python3.12
→ includes that exact Prefect version - You can pin deterministically with a digest:
prefecthq/prefect@sha256:<digest>
So, if you want the worker image to stay on a specific Prefect version, use a versioned tag (or a digest). If you use a floating tag like
:3-latest
, it will update as new 3.x releases are published. Useful links: - Docker Hub tags: prefecthq/prefect tags - Docs (Prefect 3.x) on Docker images/work pools: Docker work pool guide (default base image is
prefecthq/prefect:3-latest
unless you specify otherwise) If you meant a different image by “prefect-worker,” let me know which repository/tag you’re looking at and I can confirm its behavior.
d
My current image version is
prefecthq/prefect:3.3.5-python3.10-kubernetes
however the prefect version installed in the running container is
prefect==3.4.25
. This seems inconsistent with what you have stated and I believe it is also causing my workers to crash. Were there changes to how a worker retrieves scheduled flow runs between versions
3.3.5
and
3.4.25
? @Marvin
m
thought for 467.7 seconds
Thanks for flagging this — that’s not expected. - Versioned tags like
prefecthq/prefect:3.3.5-python3.10-kubernetes
should contain the matching Prefect package version (3.3.5). Seeing
prefect==3.4.25
in that container suggests either the tag was retagged upstream, a registry mirror served a different manifest, or something in your build/runtime upgraded Prefect inside the container. A few quick checks to confirm what’s actually running 1) Pull and inspect the tag’s digest (helps us verify if the tag moved):
Copy code
docker pull prefecthq/prefect:3.3.5-python3.10-kubernetes
docker inspect --format='{{index .RepoDigests 0}}' prefecthq/prefect:3.3.5-python3.10-kubernetes
Please share that digest (the
sha256:...
). If you have an older local copy that showed 3.3.5 previously, run the same inspect there to compare. 2) Verify the version inside a fresh container:
Copy code
docker run --rm -it prefecthq/prefect:3.3.5-python3.10-kubernetes python -c "import prefect; print(prefect.__version__)"
docker run --rm -it prefecthq/prefect:3.3.5-python3.10-kubernetes prefect version
3) Double-check nothing upgrades Prefect at runtime: - Any init/startup scripts, sidecars, or mounted requirements files that might run
pip install -U prefect
? - A custom Dockerfile layer on top of the base image that installs extras or your app’s deps? If the digest shows the tag truly points to an image with 3.4.25, that’s an issue we should fix. Sharing the digest will help us investigate. About worker behavior between 3.3.5 and 3.4.25 - There were no intentional breaking changes to how workers fetch scheduled flow runs in 3.4.x. Workers still poll the work pool scheduled runs endpoint and use concurrency leases as before. - 3.4.x did include improvements and stricter validation in a few areas (e.g., concurrency/leases, schema validation). Those should not cause a worker crash by themselves, but could surface misconfigurations more strictly. If your workers are crashing, can you share the first error stack trace lines from the worker logs? That will help us pinpoint whether this is a version mismatch, an API/validation error, or something else. Recommended mitigations right now - Pin by digest for full determinism:
Copy code
# After you obtain the digest above
prefecthq/prefect:3.3.5-python3.10-kubernetes@sha256:<digest>
- Or build a minimal custom image that pins Prefect explicitly:
Copy code
FROM prefecthq/prefect:3.3.5-python3.10
RUN pip install --no-cache-dir "prefect[kubernetes]==3.3.5"
- Or align the image tag to the version you actually want to run (e.g.,
3.4.25-python3.10-kubernetes
) if you intend to be on 3.4.x. Helpful links - Docker Hub tags: prefecthq/prefect tags - Prefect 3.4 release notes: 3.4 release notes - Work pools and Docker/Kubernetes guides: Docker work pool, Kubernetes work pool If you can share: - The tag’s current digest - Output of
prefect version
and crash logs from the worker …I can help confirm whether the tag is mis-pointed and dig into the crash specifically.
n
well what's the command you use to start the worker?
if you install stuff on top of the image, then you might have different versions
d
I think that's obvious but I'm not using any additional commands. I'm running the worker as is.
n
there's an install policy flag on
prefect worker start
, which is why i asked
well what's the command you use to start the worker?
you can pull the image and see what comes by default, so something else must be happening if you've got a different version
Copy code
» docker run --rm -it prefecthq/prefect:3.3.5-python3.10-kubernetes prefect version
Version:             3.3.5
API version:         0.8.4
Python version:      3.10.17
Git commit:          db4b7a33
Built:               Thu, Apr 17, 2025 09:29 PM
OS/Arch:             linux/aarch64
Profile:             ephemeral
Server type:         ephemeral
Pydantic version:    2.11.3
Server:
  Database:          sqlite
  SQLite version:    3.40.1
Integrations:
  prefect-kubernetes: 0.5.9
  prefect-redis:     0.2.2
d
gotcha, im double checking
My
installPolicy
is currently set to
always
.
Seems like it might be the problem
That fixed the version issue, thanks @Nate
catjam 1