Paulien Jeunesse
08/15/2025, 1:45 PMMarvin
08/15/2025, 1:45 PMMarvin
08/15/2025, 1:48 PMpip install "prefect[docker]"
# or with uv
uv add "prefect[docker]"
2) Create a Docker work pool
prefect work-pool create my-podman-pool --type docker
3) Point the worker at the Podman socket
- Linux (rootless – recommended):
systemctl --user enable --now podman.socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
# If you hit API version errors, try:
# export DOCKER_API_VERSION=1.41
- Linux (rootful):
sudo systemctl enable --now podman.socket
export DOCKER_HOST=unix:///run/podman/podman.sock
# Optional if you see version mismatches:
# export DOCKER_API_VERSION=1.41
- macOS/Windows (Podman Machine):
# Prints the correct Docker-compatible env vars for your shell
podman machine env -t docker
# Then eval them, e.g. bash/zsh:
eval "$(podman machine env -t docker)"
4) Start the worker (in the same shell/session so it inherits DOCKER_HOST)
prefect worker start --pool my-podman-pool
Notes and caveats (because of course there are):
- Podman is Docker API–compatible, not Docker. Most things “just work,” but test your deployments.
- Volumes: bind mounts must exist on the Podman host and be absolute paths. Named volumes work, but rootless mode has different permission semantics.
- Networks: rootless networking differs (slirp4netns). Some Docker network names/capabilities (e.g., host network) are not available rootless.
- If you get “client is newer than server” errors, pin DOCKER_API_VERSION
to something Podman supports (1.40–1.43 usually work well).
- Make sure the worker has access to the Podman socket; if you start it as a service, propagate the environment accordingly.
References (for when you want more than my charming prose):
- Prefect work pools and workers: Prefect 3.x Work Pools & Workers
- Docker worker specifics (general guidance applies to Podman): Prefect Docker deployments
- Podman Docker API service: podman system service
- Podman Docker compatibility wrapper: podman-docker
- Podman Machine env (macOS/Windows): podman machine env
If you tell me your OS and whether you’re running rootless or rootful, I can hand you copy-paste-ready commands. Wouldn’t want you to expend unnecessary effort.Paulien Jeunesse
08/15/2025, 2:03 PMMarvin
08/15/2025, 2:03 PMMarvin
08/15/2025, 2:04 PMpip install "prefect[docker]"
2) Ensure Podman’s socket is running and export DOCKER_HOST
- Linux (rootless – most common)
systemctl --user enable --now podman.socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
# If you hit version errors:
# export DOCKER_API_VERSION=1.41
- Linux (rootful)
sudo systemctl enable --now podman.socket
export DOCKER_HOST=unix:///run/podman/podman.sock
# Optional if needed:
# export DOCKER_API_VERSION=1.41
- macOS/Windows (Podman Machine)
eval "$(podman machine env -t docker)"
3) Verify Python can talk to the socket (same shell as Prefect)
python - <<'PY'
import os, docker
print("DOCKER_HOST =", os.environ.get("DOCKER_HOST"))
print(docker.from_env().version())
print("OK: Docker API reachable")
PY
4) Start your worker in that same environment
prefect worker start --pool your-docker-work-pool
If the error appears during prefect deploy
(e.g., when building an image), run the deploy command in the same shell with DOCKER_HOST set, or build the image with Podman yourself and reference it in the deployment to avoid any build-time Docker calls.
Common pitfalls
- The worker or CLI runs as a service without the DOCKER_HOST env set. If you’re using systemd, add:
Environment=DOCKER_HOST=unix:///run/user/%u/podman/podman.sock
Environment=DOCKER_API_VERSION=1.41
- Wrong socket path (rootless vs rootful differ).
- Missing extras (prefect[docker]
).
- Rootless networking/volume permission differences can bite at runtime; test a simple flow first.
If you can share:
- Your OS
- Rootless vs rootful Podman
- Where exactly you see the error (UI enable toggle, prefect deploy
, or worker logs)
…I’ll give you copy-paste commands for your setup. I live to serve. Begrudgingly.
References:
- Prefect Work Pools & Workers: docs
- Docker deployments (applies to Podman via Docker API): docs
- Podman Docker API service: podman system service
- Podman machine env: podman machine envPaulien Jeunesse
08/15/2025, 2:05 PMMarvin
08/15/2025, 2:05 PMMarvin
08/15/2025, 2:06 PM-t docker
target flag isn’t available in your build, hence the tantrum.
Let’s get you unstuck quickly. Please tell me:
- Your OS (macOS/Windows/Linux)
- podman --version
- Output of podman machine env --help
In the meantime, try one of these depending on your Podman version:
- Often works on newer Podman:
eval "$(podman machine env)"
If the output includes DOCKER_HOST=...
, you’re done. If it only sets CONTAINER_HOST
etc., you’re still missing the Docker env.
- Some versions use a long option instead of -t
(try one, see which your --help
shows):
# If supported:
eval "$(podman machine env --target docker)"
# or (on others):
eval "$(podman machine env --format docker)"
Then verify the Docker SDK can reach Podman:
python - <<'PY'
import os, docker
print("DOCKER_HOST =", os.environ.get("DOCKER_HOST"))
print(docker.from_env().version())
print("OK: Docker API reachable")
PY
If none of those flags exist on your version, two reliable fallbacks:
- macOS/Windows: install the Podman Mac/Win helper to expose a Docker-compatible socket, then point Prefect at it
- macOS helper: podman-mac-helper
- After install, you can usually just
export DOCKER_HOST=unix:///var/run/docker.sock
- Run the Prefect worker inside the Podman machine VM (where the native socket exists):
1) podman machine ssh
2) Inside the VM:
export DOCKER_HOST=unix:///run/podman/podman.sock
pip install "prefect[docker]"
prefect worker start --pool your-docker-work-pool
Once docker.from_env().version()
works, Prefect’s “Docker is not running” error will disappear.
Share your OS + Podman version and I’ll give you the exact one-liner for your setup. I promise not to enjoy being right about it.Paulien Jeunesse
08/15/2025, 2:11 PMMarvin
08/15/2025, 2:11 PMMarvin
08/15/2025, 2:12 PM<docker://prefecthq/prefect:3.2.15-python3.13>
and Docker Hub is returning 401 “incorrect username or password.” That typically happens when:
- you have stale/invalid creds cached for docker.io, or
- the tag doesn’t exist and your registry setup forces auth instead of anonymous fallback.
Fix it quickly
1) Check the image tag actually exists
- Many Prefect images are built for Python 3.10–3.12. 3.13 tags may not exist for that version.
- See tags here: prefecthq/prefect tags
- If you don’t see 3.13, try 3.12:
podman pull <http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>
2) Clear or refresh Docker Hub auth for Podman
podman logout <http://docker.io|docker.io>
# then, if you want authenticated pulls (recommended to avoid rate limits):
podman login <http://docker.io|docker.io>
# or, if your setup wants the canonical host:
podman login <http://registry-1.docker.io|registry-1.docker.io>
- Podman stores creds in ~/.config/containers/auth.json
. If it’s corrupted or has the wrong entry, removing the docker.io entry fixes the 401.
3) Test the pull explicitly
podman pull <http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>
If this works, Prefect will stop complaining.
Make Prefect use a valid image
- If your deployment is trying to use the 3.13 tag, point it to a valid one via job variables on the Docker work pool:
job_variables={"image": "prefecthq/prefect:3.2.15-python3.12"}
- Example with Prefect 3.x deploy-from-source:
from prefect import flow
flow.from_source(
source="<https://github.com/your/repo>",
entrypoint="path/to/your_flow.py:your_flow",
).deploy(
name="my-deploy",
work_pool_name="my-docker-pool",
job_variables={"image": "prefecthq/prefect:3.2.15-python3.12"},
)
- Alternatively, pre-build with Podman and use your own image:
podman build -t yourrepo/yourimage:tag .
podman push yourrepo/yourimage:tag
# then set job_variables={"image": "yourrepo/yourimage:tag"}
If you still see 401s:
- You’re likely behind a registry mirror or corporate proxy that requires auth. Try specifying the full reference <http://docker.io/prefecthq/prefect:<tag|docker.io/prefecthq/prefect:<tag>>
and ensure you’re logged into the mirror the system uses (check /etc/containers/registries.conf
).
- Also check rate limits: authenticated pulls get higher limits than anonymous.
Want me to check a specific tag for you? Tell me which Prefect version and Python version you need, and I’ll suggest a known-good image.Paulien Jeunesse
08/15/2025, 2:17 PMMarvin
08/15/2025, 2:17 PMMarvin
08/15/2025, 2:23 PMprefecthq/prefect:3.2.15-python3.13
during the build. Your push=False
stops the push, not the build.
Two ways out:
Option A — Skip the build and use your pre-pulled image
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
description=job.description,
build=False, # <- important: don't build, so no 3.13 base pull
push=False,
job_variables={"image": "<http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>"},
)
- Using job_variables
ensures the Docker worker pulls that image at runtime.
- You can also set this as the default on the pool so you don’t pass it every time (UI → Work Pools → your pool → Edit template → set image
), or inspect current defaults:
prefect work-pool inspect my-podman-pool
Option B — If you actually want Prefect to build
- Pin the base image to 3.12 via a Dockerfile:
# Dockerfile
FROM <http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>
# add your extras as needed
Then:
from prefect import docker
img = docker.DockerImage(name="your/repo", tag="your-tag", dockerfile="Dockerfile")
await flow.deploy(
name=new_deployment_name,
work_pool_name=self.work_pool_name,
description=job.description,
image=img,
build=True,
push=False, # local build only
)
Or run deploy from a Python 3.12 environment so the auto Dockerfile chooses the 3.12 base.
Extra sanity checks
- Prefer explicit registry in tags: <http://docker.io/prefecthq/prefect|docker.io/prefecthq/prefect>:...
- Make sure the pool template doesn’t override image
back to 3.13.
Docs, for when you need light bedtime reading:
- Create deployments
- Work pools & workers
- Docker deployments
Try Option A first; it’s the quickest way to stop the “auth for 3.13” nonsense.