Del
10/24/2025, 4:27 PMMarvin
10/24/2025, 4:27 PMMarvin
10/24/2025, 4:31 PMprefecthq/prefect.
- Floating tags move forward as new releases come out:
- :latest → latest stable Prefect of the latest major
- :3-latest → latest Prefect 3.x
- :3-python3.12 (and similar) → latest Prefect 3.x for that Python minor
- Pinned tags are versioned and do not change:
- :3.1.2 or :3.1.2-python3.12 → includes that exact Prefect version
- You can pin deterministically with a digest: prefecthq/prefect@sha256:<digest>
So, if you want the worker image to stay on a specific Prefect version, use a versioned tag (or a digest). If you use a floating tag like :3-latest, it will update as new 3.x releases are published.
Useful links:
- Docker Hub tags: prefecthq/prefect tags
- Docs (Prefect 3.x) on Docker images/work pools: Docker work pool guide (default base image is prefecthq/prefect:3-latest unless you specify otherwise)
If you meant a different image by “prefect-worker,” let me know which repository/tag you’re looking at and I can confirm its behavior.Del
10/24/2025, 4:40 PMprefecthq/prefect:3.3.5-python3.10-kubernetes however the prefect version installed in the running container is prefect==3.4.25. This seems inconsistent with what you have stated and I believe it is also causing my workers to crash.
Were there changes to how a worker retrieves scheduled flow runs between versions 3.3.5 and 3.4.25 ?
@MarvinMarvin
10/24/2025, 4:40 PMMarvin
10/24/2025, 4:48 PMprefecthq/prefect:3.3.5-python3.10-kubernetes should contain the matching Prefect package version (3.3.5). Seeing prefect==3.4.25 in that container suggests either the tag was retagged upstream, a registry mirror served a different manifest, or something in your build/runtime upgraded Prefect inside the container.
A few quick checks to confirm what’s actually running
1) Pull and inspect the tag’s digest (helps us verify if the tag moved):
docker pull prefecthq/prefect:3.3.5-python3.10-kubernetes
docker inspect --format='{{index .RepoDigests 0}}' prefecthq/prefect:3.3.5-python3.10-kubernetes
Please share that digest (the sha256:...). If you have an older local copy that showed 3.3.5 previously, run the same inspect there to compare.
2) Verify the version inside a fresh container:
docker run --rm -it prefecthq/prefect:3.3.5-python3.10-kubernetes python -c "import prefect; print(prefect.__version__)"
docker run --rm -it prefecthq/prefect:3.3.5-python3.10-kubernetes prefect version
3) Double-check nothing upgrades Prefect at runtime:
- Any init/startup scripts, sidecars, or mounted requirements files that might run pip install -U prefect?
- A custom Dockerfile layer on top of the base image that installs extras or your app’s deps?
If the digest shows the tag truly points to an image with 3.4.25, that’s an issue we should fix. Sharing the digest will help us investigate.
About worker behavior between 3.3.5 and 3.4.25
- There were no intentional breaking changes to how workers fetch scheduled flow runs in 3.4.x. Workers still poll the work pool scheduled runs endpoint and use concurrency leases as before.
- 3.4.x did include improvements and stricter validation in a few areas (e.g., concurrency/leases, schema validation). Those should not cause a worker crash by themselves, but could surface misconfigurations more strictly.
If your workers are crashing, can you share the first error stack trace lines from the worker logs? That will help us pinpoint whether this is a version mismatch, an API/validation error, or something else.
Recommended mitigations right now
- Pin by digest for full determinism:
# After you obtain the digest above
prefecthq/prefect:3.3.5-python3.10-kubernetes@sha256:<digest>
- Or build a minimal custom image that pins Prefect explicitly:
FROM prefecthq/prefect:3.3.5-python3.10
RUN pip install --no-cache-dir "prefect[kubernetes]==3.3.5"
- Or align the image tag to the version you actually want to run (e.g., 3.4.25-python3.10-kubernetes) if you intend to be on 3.4.x.
Helpful links
- Docker Hub tags: prefecthq/prefect tags
- Prefect 3.4 release notes: 3.4 release notes
- Work pools and Docker/Kubernetes guides: Docker work pool, Kubernetes work pool
If you can share:
- The tag’s current digest
- Output of prefect version and crash logs from the worker
…I can help confirm whether the tag is mis-pointed and dig into the crash specifically.Nate
10/24/2025, 4:54 PMNate
10/24/2025, 4:55 PMDel
10/24/2025, 4:56 PMNate
10/24/2025, 4:58 PMprefect worker start, which is why i asked
well what's the command you use to start the worker?you can pull the image and see what comes by default, so something else must be happening if you've got a different version
» docker run --rm -it prefecthq/prefect:3.3.5-python3.10-kubernetes prefect version
Version: 3.3.5
API version: 0.8.4
Python version: 3.10.17
Git commit: db4b7a33
Built: Thu, Apr 17, 2025 09:29 PM
OS/Arch: linux/aarch64
Profile: ephemeral
Server type: ephemeral
Pydantic version: 2.11.3
Server:
Database: sqlite
SQLite version: 3.40.1
Integrations:
prefect-kubernetes: 0.5.9
prefect-redis: 0.2.2Del
10/24/2025, 4:59 PMDel
10/24/2025, 5:01 PMinstallPolicy is currently set to always.Del
10/24/2025, 5:01 PMDel
10/24/2025, 5:03 PM