<@ULVA73B9P> How can I use a docker work pool but ...
# ask-marvin
p
@Marvin How can I use a docker work pool but with podman?
m
thought for 210.6 seconds
Short answer: run a Docker worker, point it at Podman’s Docker-compatible socket via DOCKER_HOST, and carry on. Here’s the minimal setup. 1) Install the Docker extra
Copy code
pip install "prefect[docker]"
# or with uv
uv add "prefect[docker]"
2) Create a Docker work pool
Copy code
prefect work-pool create my-podman-pool --type docker
3) Point the worker at the Podman socket - Linux (rootless – recommended):
Copy code
systemctl --user enable --now podman.socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
# If you hit API version errors, try:
# export DOCKER_API_VERSION=1.41
- Linux (rootful):
Copy code
sudo systemctl enable --now podman.socket
export DOCKER_HOST=unix:///run/podman/podman.sock
# Optional if you see version mismatches:
# export DOCKER_API_VERSION=1.41
- macOS/Windows (Podman Machine):
Copy code
# Prints the correct Docker-compatible env vars for your shell
podman machine env -t docker
# Then eval them, e.g. bash/zsh:
eval "$(podman machine env -t docker)"
4) Start the worker (in the same shell/session so it inherits DOCKER_HOST)
Copy code
prefect worker start --pool my-podman-pool
Notes and caveats (because of course there are): - Podman is Docker API–compatible, not Docker. Most things “just work,” but test your deployments. - Volumes: bind mounts must exist on the Podman host and be absolute paths. Named volumes work, but rootless mode has different permission semantics. - Networks: rootless networking differs (slirp4netns). Some Docker network names/capabilities (e.g., host network) are not available rootless. - If you get “client is newer than server” errors, pin
DOCKER_API_VERSION
to something Podman supports (1.40–1.43 usually work well). - Make sure the worker has access to the Podman socket; if you start it as a service, propagate the environment accordingly. References (for when you want more than my charming prose): - Prefect work pools and workers: Prefect 3.x Work Pools &amp; Workers - Docker worker specifics (general guidance applies to Podman): Prefect Docker deployments - Podman Docker API service: podman system service - Podman Docker compatibility wrapper: podman-docker - Podman Machine env (macOS/Windows): podman machine env If you tell me your OS and whether you’re running rootless or rootful, I can hand you copy-paste-ready commands. Wouldn’t want you to expend unnecessary effort.
p
@Marvin I get I get Error enabling deployment: This error is often thrown because Docker is not running. Please ensure Docker is running.
m
thought for 47.2 seconds
That error usually means the Docker SDK can’t reach any Docker socket. With Podman, you need to point Prefect at Podman’s Docker-compatible socket. A few quick checks/fixes: 1) Install Docker extras (yes, even with Podman)
Copy code
pip install "prefect[docker]"
2) Ensure Podman’s socket is running and export DOCKER_HOST - Linux (rootless – most common)
Copy code
systemctl --user enable --now podman.socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock
# If you hit version errors:
# export DOCKER_API_VERSION=1.41
- Linux (rootful)
Copy code
sudo systemctl enable --now podman.socket
export DOCKER_HOST=unix:///run/podman/podman.sock
# Optional if needed:
# export DOCKER_API_VERSION=1.41
- macOS/Windows (Podman Machine)
Copy code
eval "$(podman machine env -t docker)"
3) Verify Python can talk to the socket (same shell as Prefect)
Copy code
python - <<'PY'
import os, docker
print("DOCKER_HOST =", os.environ.get("DOCKER_HOST"))
print(docker.from_env().version())
print("OK: Docker API reachable")
PY
4) Start your worker in that same environment
Copy code
prefect worker start --pool your-docker-work-pool
If the error appears during
prefect deploy
(e.g., when building an image), run the deploy command in the same shell with DOCKER_HOST set, or build the image with Podman yourself and reference it in the deployment to avoid any build-time Docker calls. Common pitfalls - The worker or CLI runs as a service without the DOCKER_HOST env set. If you’re using systemd, add:
Copy code
Environment=DOCKER_HOST=unix:///run/user/%u/podman/podman.sock
Environment=DOCKER_API_VERSION=1.41
- Wrong socket path (rootless vs rootful differ). - Missing extras (
prefect[docker]
). - Rootless networking/volume permission differences can bite at runtime; test a simple flow first. If you can share: - Your OS - Rootless vs rootful Podman - Where exactly you see the error (UI enable toggle,
prefect deploy
, or worker logs) …I’ll give you copy-paste commands for your setup. I live to serve. Begrudgingly. References: - Prefect Work Pools & Workers: docs - Docker deployments (applies to Podman via Docker API): docs - Podman Docker API service: podman system service - Podman machine env: podman machine env
p
@Marvin just eval "$(podman machine env -t docker)" Error: unknown shorthand flag: 't' in -t See 'podman machine --help' it does not work
m
thought for 91.6 seconds
Ah, Podman’s flags moved around between versions. The
-t docker
target flag isn’t available in your build, hence the tantrum. Let’s get you unstuck quickly. Please tell me: - Your OS (macOS/Windows/Linux) -
podman --version
- Output of
podman machine env --help
In the meantime, try one of these depending on your Podman version: - Often works on newer Podman:
Copy code
eval "$(podman machine env)"
If the output includes
DOCKER_HOST=...
, you’re done. If it only sets
CONTAINER_HOST
etc., you’re still missing the Docker env. - Some versions use a long option instead of
-t
(try one, see which your
--help
shows):
Copy code
# If supported:
eval "$(podman machine env --target docker)"
# or (on others):
eval "$(podman machine env --format docker)"
Then verify the Docker SDK can reach Podman:
Copy code
python - <<'PY'
import os, docker
print("DOCKER_HOST =", os.environ.get("DOCKER_HOST"))
print(docker.from_env().version())
print("OK: Docker API reachable")
PY
If none of those flags exist on your version, two reliable fallbacks: - macOS/Windows: install the Podman Mac/Win helper to expose a Docker-compatible socket, then point Prefect at it - macOS helper: podman-mac-helper - After install, you can usually just
Copy code
export DOCKER_HOST=unix:///var/run/docker.sock
- Run the Prefect worker inside the Podman machine VM (where the native socket exists): 1)
podman machine ssh
2) Inside the VM:
Copy code
export DOCKER_HOST=unix:///run/podman/podman.sock
     pip install "prefect[docker]"
     prefect worker start --pool your-docker-work-pool
Once
docker.from_env().version()
works, Prefect’s “Docker is not running” error will disappear. Share your OS + Podman version and I’ll give you the exact one-liner for your setup. I promise not to enjoy being right about it.
p
@Marvin I get Error enabling deployment: creating build container: internal error: unable to copy from source docker://prefecthq/prefect:3.2.15-python3.13: initializing source docker://prefecthq/prefect:3.2.15-python3.13: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password now (seems like it is working?)
m
thought for 70.5 seconds
Good news: you’re talking to Podman now. Bad news: it’s trying to pull from Docker Hub with bad credentials. What that error means - Podman is pulling
<docker://prefecthq/prefect:3.2.15-python3.13>
and Docker Hub is returning 401 “incorrect username or password.” That typically happens when: - you have stale/invalid creds cached for docker.io, or - the tag doesn’t exist and your registry setup forces auth instead of anonymous fallback. Fix it quickly 1) Check the image tag actually exists - Many Prefect images are built for Python 3.10–3.12. 3.13 tags may not exist for that version. - See tags here: prefecthq/prefect tags - If you don’t see 3.13, try 3.12:
Copy code
podman pull <http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>
2) Clear or refresh Docker Hub auth for Podman
Copy code
podman logout <http://docker.io|docker.io>
# then, if you want authenticated pulls (recommended to avoid rate limits):
podman login <http://docker.io|docker.io>
# or, if your setup wants the canonical host:
podman login <http://registry-1.docker.io|registry-1.docker.io>
- Podman stores creds in
~/.config/containers/auth.json
. If it’s corrupted or has the wrong entry, removing the docker.io entry fixes the 401. 3) Test the pull explicitly
Copy code
podman pull <http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>
If this works, Prefect will stop complaining. Make Prefect use a valid image - If your deployment is trying to use the 3.13 tag, point it to a valid one via job variables on the Docker work pool:
Copy code
job_variables={"image": "prefecthq/prefect:3.2.15-python3.12"}
- Example with Prefect 3.x deploy-from-source:
Copy code
from prefect import flow
flow.from_source(
    source="<https://github.com/your/repo>",
    entrypoint="path/to/your_flow.py:your_flow",
).deploy(
    name="my-deploy",
    work_pool_name="my-docker-pool",
    job_variables={"image": "prefecthq/prefect:3.2.15-python3.12"},
)
- Alternatively, pre-build with Podman and use your own image:
Copy code
podman build -t yourrepo/yourimage:tag .
podman push yourrepo/yourimage:tag
# then set job_variables={"image": "yourrepo/yourimage:tag"}
If you still see 401s: - You’re likely behind a registry mirror or corporate proxy that requires auth. Try specifying the full reference
<http://docker.io/prefecthq/prefect:<tag|docker.io/prefecthq/prefect:<tag>>
and ensure you’re logged into the mirror the system uses (check
/etc/containers/registries.conf
). - Also check rate limits: authenticated pulls get higher limits than anonymous. Want me to check a specific tag for you? Tell me which Prefect version and Python version you need, and I’ll suggest a known-good image.
p
@Marvin I still get Error enabling deployment: creating build container: internal error: unable to copy from source docker://prefecthq/prefect:3.2.15-python3.13: initializing source docker://prefecthq/prefect:3.2.15-python3.13: unable to retrieve auth token: invalid username/password: unauthorized: incorrect username or password but when I do podman pull docker.io/prefecthq/prefect:3.2.15-python3.12 it works. ALso i do that await flow.deploy( name=new_deployment_name, work_pool_name=self.work_pool_name, # Uses my-podman-pool description=job.description, image="prefecthq/prefect:3.2.15-python3.12", # Use the working Prefect image push=False, # Don't push to registry for local development )
m
thought for 346.4 seconds
You’re still hitting a build step. Prefect is auto-generating a Dockerfile and picking a base image that matches your local Python (3.13), so it tries to pull
prefecthq/prefect:3.2.15-python3.13
during the build. Your
push=False
stops the push, not the build. Two ways out: Option A — Skip the build and use your pre-pulled image
Copy code
await flow.deploy(
    name=new_deployment_name,
    work_pool_name=self.work_pool_name,
    description=job.description,
    build=False,  # <- important: don't build, so no 3.13 base pull
    push=False,
    job_variables={"image": "<http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>"},
)
- Using
job_variables
ensures the Docker worker pulls that image at runtime. - You can also set this as the default on the pool so you don’t pass it every time (UI → Work Pools → your pool → Edit template → set
image
), or inspect current defaults:
Copy code
prefect work-pool inspect my-podman-pool
Option B — If you actually want Prefect to build - Pin the base image to 3.12 via a Dockerfile:
Copy code
# Dockerfile
FROM <http://docker.io/prefecthq/prefect:3.2.15-python3.12|docker.io/prefecthq/prefect:3.2.15-python3.12>
# add your extras as needed
Then:
Copy code
from prefect import docker
img = docker.DockerImage(name="your/repo", tag="your-tag", dockerfile="Dockerfile")

await flow.deploy(
    name=new_deployment_name,
    work_pool_name=self.work_pool_name,
    description=job.description,
    image=img,
    build=True,
    push=False,  # local build only
)
Or run deploy from a Python 3.12 environment so the auto Dockerfile chooses the 3.12 base. Extra sanity checks - Prefer explicit registry in tags:
<http://docker.io/prefecthq/prefect|docker.io/prefecthq/prefect>:...
- Make sure the pool template doesn’t override
image
back to 3.13. Docs, for when you need light bedtime reading: - Create deployments - Work pools &amp; workers - Docker deployments Try Option A first; it’s the quickest way to stop the “auth for 3.13” nonsense.