my dockerfile is pretty simple:

FROM <http://docker.io/prefecthq/prefect:3-latest|docker.io/prefecthq/prefect:3-latest>
COPY ./requirements.txt .
RUN pip install -r requirements.txt

and of course I need to update and build a new one whenever I require new libraries

I use the

prefect.yaml

file to manage my deployments, whenever I create a new deployment it asks me if I want to build a docker image, I havent tried it... but maybe that automates the process a bit more idk

anyway, good luck!

Great question — and congrats on getting your self-hosted setup running! In Prefect 3.x, dependencies aren’t installed “by Prefect” at run time; they come from the environment your worker uses to execute your flow. How you include them depends on your work pool type: - Process worker: Your flow runs in the same Python environment as the worker. Dependencies must already be installed there (e.g., in the container/venv where the worker runs). Prefect won’t pip install per-deployment for Process workers. - Docker or Kubernetes worker: Your flow runs in a container. You specify an image for the deployment that has your dependencies baked in. Prefect can use an existing image or build one for you during deployment. A couple of quick questions to steer you the right way: - Which work pool type are you using for

local-pool

— Process or Docker? - Where does your

requirements.txt

live in your repo? If you’re using a Docker/Kubernetes worker, here are two simple ways to add dependencies with your current code-first pattern: 1) Use a prebuilt image (already contains your requirements) - Build and push your image (e.g., extending

prefecthq/prefect

and installing your requirements) - Then reference it in `.deploy(...)`:

from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source=f"https://{access_token}@github.com/tom/{repo}.git",
        entrypoint=entrypoint,  # e.g. "flows/my_flow.py:my_flow"
    ).deploy(
        name=pipeline_name,
        work_pool_name="docker-pool",
        image="<http://ghcr.io/your-org/your-project:2025-01-01|ghcr.io/your-org/your-project:2025-01-01>",  # your image with deps
    )

2) Let Prefect build the image for you during deployment - Put a

requirements.txt

(and/or a Dockerfile) in your repo - Use `prefect.docker.DockerImage`; Prefect will build and push when you call `deploy()`:

from prefect import flow
from prefect.docker import DockerImage

if __name__ == "__main__":
    image = DockerImage(
        name="<http://ghcr.io/your-org/your-project|ghcr.io/your-org/your-project>",
        tag="latest",
        dockerfile="auto",  # synthesize a simple Dockerfile and try to install deps found in the repo
        # you can pass extra docker build kwargs via **build_kwargs if needed
    )

    flow.from_source(
        source=f"https://{access_token}@github.com/tom/{repo}.git",
        entrypoint=entrypoint,
    ).deploy(
        name=pipeline_name,
        work_pool_name="docker-pool",
        image=image,
        build=True,   # default True
        push=True,    # default True
    )

Notes for the “auto” Dockerfile path: - Put

requirements.txt

at the repo root (or include a

pyproject.toml/setup.cfg

) so the auto build can detect and install your deps. - The machine running

.deploy(...)

must have access to a Docker daemon and to your registry (docker login). If you prefer to control the Dockerfile explicitly, here’s a minimal example you can keep in your repo:

# Dockerfile
FROM prefecthq/prefect:3-python3.11
WORKDIR /opt/prefect
COPY . /opt/prefect
RUN pip install --no-cache-dir -r requirements.txt

Then:

from prefect.docker import DockerImage

image = DockerImage(
    name="<http://ghcr.io/your-org/your-project|ghcr.io/your-org/your-project>",
    tag="0.1.0",
    dockerfile="Dockerfile",
)

flow.from_source(...).deploy(
    name="my-deployment",
    work_pool_name="docker-pool",
    image=image,
    build=True,
    push=True,
)

If you’re on a Process worker - Install your dependencies into the environment where the worker runs (e.g., bake them into the worker’s container if you’re running the worker in Docker, or install them in the venv if running locally). - Your deployment code stays the same (you generally won’t specify an image for Process workers). YAML vs code-first - You do not need to switch to YAML. Your current code-first approach fully supports specifying images and building them as shown above. If you can confirm: - your work pool type (Process vs Docker/Kubernetes), - and where your

requirements.txt

lives,

I’ll tailor the exact snippet you need and call out any worker/pool settings to double-check.

Awesome — thanks for the details. Short answers first: - To switch from a Process pool to Docker, you’ll create a Docker work pool and start a Docker worker in your docker-compose. The worker container must have access to the host Docker daemon (mount

/var/run/docker.sock

) and the

prefect[docker]

extra installed. - When you call

.deploy(..., image=..., build=True, push=True)

, Prefect builds the image locally and pushes it to the registry implied by the image name you provide (e.g.,

youruser/yourimage

→ Docker Hub,

<http://ghcr.io/org/image|ghcr.io/org/image>

→ GHCR,

<http://1234567890.dkr.ecr.region.amazonaws.com/repo:tag|1234567890.dkr.ecr.region.amazonaws.com/repo:tag>

→ AWS ECR). You must be docker-logged-in on the machine running

deploy()

. Your Docker worker will then pull that image at run time. Step-by-step to switch to a Docker pool 1) Create a Docker work pool (one-time) - From anywhere that can reach your Prefect Server API:

prefect work-pool create docker-pool --type docker

You can also create it in the UI if you prefer. 2) Add a Docker worker service to your docker-compose - Key points: - Use a Prefect image - Ensure

prefect[docker]

is installed in the worker container - Mount the Docker socket so the worker can start job containers - Point the worker at your Server API and the docker-pool you created Example snippet you can adapt:

services:
  prefect-server:
    image: prefecthq/prefect:3-latest
    # ... your existing server config ...

  prefect-worker-docker:
    image: prefecthq/prefect:3-python3.11
    # Install the docker extra, then start the worker
    command: >
      bash -c "pip install --no-cache-dir 'prefect[docker]' &&
               prefect worker start -p docker-pool"
    environment:
      # If self-hosting Prefect Server:
      PREFECT_API_URL: <http://prefect-server:4200/api>
      PREFECT_LOGGING_LEVEL: INFO
      # If using Prefect Cloud, you'd set PREFECT_API_URL + PREFECT_API_KEY instead
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - prefect-server
    restart: unless-stopped

Tip: If you prefer a cleaner container start (no pip on boot), build a small image:

# Dockerfile for the worker
FROM prefecthq/prefect:3-python3.11
RUN pip install --no-cache-dir 'prefect[docker]'

Then use that image in your compose and set the command to:

command: prefect worker start -p docker-pool

3) Update your deployment to specify an image (build or prebuilt) Option A: Use a prebuilt image (already contains your deps)

flow.from_source(
    source=f"https://{access_token}@github.com/tom/{repo}.git",
    entrypoint=entrypoint,
).deploy(
    name=pipeline_name,
    work_pool_name="docker-pool",
    image="<http://ghcr.io/your-org/your-project:0.1.0|ghcr.io/your-org/your-project:0.1.0>",  # or dockerhub: youruser/yourimage:tag
)

Option B: Let Prefect build and push the image during deploy - Put a

requirements.txt

(and/or Dockerfile) in your repo - Prefect will build and push when you call

deploy()

if you pass a

DockerImage

and leave

build=True

,

push=True

(defaults)

from prefect.docker import DockerImage

image = DockerImage(
    name="youruser/yourimage",  # Docker Hub
    tag="latest",
    dockerfile="auto",          # Prefect synthesizes a simple Dockerfile and
                                # installs requirements it finds in the repo
)

flow.from_source(
    source=f"https://{access_token}@github.com/tom/{repo}.git",
    entrypoint=entrypoint,
).deploy(
    name=pipeline_name,
    work_pool_name="docker-pool",
    image=image,
    build=True,
    push=True,
)

Where does the image get pushed? - To the registry implied by the

image

name: -

youruser/yourimage:tag

→ Docker Hub -

<http://ghcr.io/org/repo:tag|ghcr.io/org/repo:tag>

→ GitHub Container Registry -

<http://1234567890.dkr.ecr.region.amazonaws.com/repo:tag|1234567890.dkr.ecr.region.amazonaws.com/repo:tag>

→ AWS ECR - Make sure the machine running

.deploy(...)

has Docker and is logged into that registry (

docker login ...

).

- The Docker worker pulls that image at runtime. If it’s a private registry, ensure the host daemon the worker uses is logged in. Since you’re mounting

/var/run/docker.sock

, the worker will use the host’s Docker credentials. Notes if you keep a Process worker too - Process workers run flows in their own environment. They won’t install your dependencies per-deployment; you’d install deps into the worker’s environment or image. You can run both pools side-by-side if you want. If you share: - which registry you want to use (Docker Hub, GHCR, ECR), - where

requirements.txt

lives in your repo, I can give you a tailored compose + deploy snippet for that setup.