<@ULVA73B9P> if I need to switch my local deployme...
# ask-marvin
c
@Marvin if I need to switch my local deployment to Docker based deployment, how do I do this?
m
thought for 434.6 seconds
Great question. Here’s a concise path to move from a local/process deployment to a Docker-based deployment in Prefect 3.x. I’ll assume: - You’ll run a Docker worker on the same host (Podman is allowed, see notes below) - You need your flows to access host file paths - Your image will be stored in JFrog Artifactory High-level steps 1) Install the Docker integration - On the machine that will run the worker:
Copy code
pip install "prefect[docker]"
# or
pip install prefect-docker
- Docs: Prefect Docker collection docs - Concepts: Workers and Work Pools 2) Build and push a Docker image (to Artifactory) - Example Dockerfile:
Copy code
FROM python:3.11-slim

# Install system deps if needed
# RUN apt-get update && apt-get install -y libpq-dev gcc ...

# Install your code and dependencies
WORKDIR /app
COPY pyproject.toml poetry.lock* requirements.txt* /app/
# If using requirements.txt:
RUN pip install --no-cache-dir -r requirements.txt
# If using Poetry:
# RUN pip install poetry && poetry config virtualenvs.create false && poetry install --no-interaction --no-ansi

# Copy your source
COPY . /app

# Optional: set a default working directory
ENV PYTHONPATH=/app
- Tag/push to Artifactory:
Copy code
docker login <your-artifactory-domain>  # or: podman login <your-artifactory-domain>
docker build -t <your-artifactory-domain>/<repo>/my-flow:1.0.0 .
docker push <your-artifactory-domain>/<repo>/my-flow:1.0.0
- JFrog info: JFrog Artifactory Docker registry 3) Create a Docker work pool - Create a new pool for Docker runs:
Copy code
prefect work-pool create my-docker-pool --type docker
4) Start a Docker worker for that pool - If using Docker Engine:
Copy code
prefect worker start -p my-docker-pool
- If using Podman (Docker API-compatible): - Rootless (typical):
Copy code
systemctl --user enable --now podman.socket
  export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock
  prefect worker start -p my-docker-pool
- Rootful (system service):
Copy code
systemctl enable --now podman.socket
  export DOCKER_HOST=unix:///run/podman/podman.sock
  prefect worker start -p my-docker-pool
- Podman service docs: podman system service 5) Update your deployment to target the Docker pool and mount host paths Option A — prefect.yaml (recommended if you use prefect deploy) - In your project’s prefect.yaml, define a deployment with Docker job variables. Example:
Copy code
deployments:
- name: docker
  entrypoint: flows/my_flow.py:my_flow   # adjust to your file and flow name
  work_pool:
    name: my-docker-pool
    job_variables:
      image: "<your-artifactory-domain>/<repo>/my-flow:1.0.0"
      env:
        PREFECT_LOGGING_LEVEL: "INFO"
      # Mount host paths your flow needs
      volumes:
        - "/host/data:/data:rw"
        - "/host/config:/app/config:ro"
      # For Podman on SELinux hosts, you may need :Z
      # volumes:
      #   - "/host/data:/data:rw,Z"
- Apply:
Copy code
prefect deploy
Option B — Python API - If you prefer to create the deployment from Python:
Copy code
from prefect import flow

@flow
def my_flow():
    ...

# If your code is local, you can still deploy from the local project:
my_flow.from_source(
    source=".",
    entrypoint="flows/my_flow.py:my_flow",
).deploy(
    name="docker",
    work_pool_name="my-docker-pool",
    job_variables={
        "image": "<your-artifactory-domain>/<repo>/my-flow:1.0.0",
        "env": {"PREFECT_LOGGING_LEVEL": "INFO"},
        "volumes": ["/host/data:/data:rw", "/host/config:/app/config:ro"],
        # For Podman/SELinux: "/host/data:/data:rw,Z"
    },
)
6) Test the deployment - Trigger a run:
Copy code
prefect deployment run "my_flow/docker"
Notes and gotchas
- Artifactory auth: Make sure the worker host is logged in to your registry (
docker login
or
podman login
) so it can pull the image. - Host file access: Use
volumes
to map the exact host paths your flow needs into the container at the same paths your code expects. For Podman with SELinux, append
:Z
to volume mounts. - Where results land: If your flow writes outputs to local paths, ensure those paths are mounted, otherwise the container’s filesystem is ephemeral and data will be lost after the run. - Process worker to Docker worker: Once you create the Docker pool and start its worker, just point your deployment to that pool. You can stop your process worker after you validate the Docker runs. Would you like me to: - Draft a minimal prefect.yaml for your project? - Provide a sample Dockerfile tailored to your dependencies? - Validate the exact host paths you need mounted and the correct Artifactory image URL?