Cullen Baker
10/20/2025, 9:47 PMMarvin
10/20/2025, 9:48 PMMarvin
10/20/2025, 9:55 PMpip install "prefect[docker]"
# or
pip install prefect-docker
- Docs: Prefect Docker collection docs
- Concepts: Workers and Work Pools
2) Build and push a Docker image (to Artifactory)
- Example Dockerfile:
FROM python:3.11-slim
# Install system deps if needed
# RUN apt-get update && apt-get install -y libpq-dev gcc ...
# Install your code and dependencies
WORKDIR /app
COPY pyproject.toml poetry.lock* requirements.txt* /app/
# If using requirements.txt:
RUN pip install --no-cache-dir -r requirements.txt
# If using Poetry:
# RUN pip install poetry && poetry config virtualenvs.create false && poetry install --no-interaction --no-ansi
# Copy your source
COPY . /app
# Optional: set a default working directory
ENV PYTHONPATH=/app
- Tag/push to Artifactory:
docker login <your-artifactory-domain> # or: podman login <your-artifactory-domain>
docker build -t <your-artifactory-domain>/<repo>/my-flow:1.0.0 .
docker push <your-artifactory-domain>/<repo>/my-flow:1.0.0
- JFrog info: JFrog Artifactory Docker registry
3) Create a Docker work pool
- Create a new pool for Docker runs:
prefect work-pool create my-docker-pool --type docker
4) Start a Docker worker for that pool
- If using Docker Engine:
prefect worker start -p my-docker-pool
- If using Podman (Docker API-compatible):
- Rootless (typical):
systemctl --user enable --now podman.socket
export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock
prefect worker start -p my-docker-pool
- Rootful (system service):
systemctl enable --now podman.socket
export DOCKER_HOST=unix:///run/podman/podman.sock
prefect worker start -p my-docker-pool
- Podman service docs: podman system service
5) Update your deployment to target the Docker pool and mount host paths
Option A — prefect.yaml (recommended if you use prefect deploy)
- In your project’s prefect.yaml, define a deployment with Docker job variables. Example:
deployments:
- name: docker
entrypoint: flows/my_flow.py:my_flow # adjust to your file and flow name
work_pool:
name: my-docker-pool
job_variables:
image: "<your-artifactory-domain>/<repo>/my-flow:1.0.0"
env:
PREFECT_LOGGING_LEVEL: "INFO"
# Mount host paths your flow needs
volumes:
- "/host/data:/data:rw"
- "/host/config:/app/config:ro"
# For Podman on SELinux hosts, you may need :Z
# volumes:
# - "/host/data:/data:rw,Z"
- Apply:
prefect deploy
Option B — Python API
- If you prefer to create the deployment from Python:
from prefect import flow
@flow
def my_flow():
...
# If your code is local, you can still deploy from the local project:
my_flow.from_source(
source=".",
entrypoint="flows/my_flow.py:my_flow",
).deploy(
name="docker",
work_pool_name="my-docker-pool",
job_variables={
"image": "<your-artifactory-domain>/<repo>/my-flow:1.0.0",
"env": {"PREFECT_LOGGING_LEVEL": "INFO"},
"volumes": ["/host/data:/data:rw", "/host/config:/app/config:ro"],
# For Podman/SELinux: "/host/data:/data:rw,Z"
},
)
6) Test the deployment
- Trigger a run:
prefect deployment run "my_flow/docker"
Notes and gotchasMarvin
10/20/2025, 9:55 PMdocker login or podman login) so it can pull the image.
- Host file access: Use volumes to map the exact host paths your flow needs into the container at the same paths your code expects. For Podman with SELinux, append :Z to volume mounts.
- Where results land: If your flow writes outputs to local paths, ensure those paths are mounted, otherwise the container’s filesystem is ephemeral and data will be lost after the run.
- Process worker to Docker worker: Once you create the Docker pool and start its worker, just point your deployment to that pool. You can stop your process worker after you validate the Docker runs.
Would you like me to:
- Draft a minimal prefect.yaml for your project?
- Provide a sample Dockerfile tailored to your dependencies?
- Validate the exact host paths you need mounted and the correct Artifactory image URL?