Shubham
12/25/2025, 2:53 PMMarvin
12/25/2025, 2:53 PMMarvin
12/25/2025, 2:58 PMflow.from_source(...) to point Prefect at your repo
- Let the official Prefect image install extra packages at runtime via the EXTRA_PIP_PACKAGES env var (handled by the image’s entrypoint)
Example:
from prefect import flow
@flow(log_prints=True)
def my_flow():
import pandas
print("Pandas version:", pandas.__version__)
if __name__ == "__main__":
flow.from_source(
source="<https://github.com/your-org/your-repo.git>", # or a GitRepository with credentials
entrypoint="flows.py:my_flow",
).deploy(
name="git-runtime-install",
work_pool_name="my-docker-pool",
job_variables={
"env": {
"EXTRA_PIP_PACKAGES": "pandas boto3"
}
},
)
Notes:
- For private repos, use prefect.runner.storage.GitRepository with credentials. See Store flow code guide
- EXTRA_PIP_PACKAGES works when you use Prefect’s official images and don’t override the container entrypoint
- Prefer listing packages explicitly; using -r requirements.txt can be brittle because package install may occur before code is pulled
2) Bake your code and dependencies into a Docker image
- Best for reproducibility, faster startup, and complex/native deps
- You can let Prefect generate a Dockerfile from your project or point to your own Dockerfile
Minimal example using an auto-generated Dockerfile:
from prefect import flow
@flow
def my_flow():
import pandas
print("works!")
if __name__ == "__main__":
my_flow.deploy(
name="image-baked",
work_pool_name="my-docker-pool",
image="my-registry/my-image:latest",
build=True, # build an image from your current directory
push=True, # push to your registry
)
Custom Dockerfile example:
# Dockerfile
FROM prefecthq/prefect:3-python3.11
COPY requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
COPY . /opt/prefect/flows/
WORKDIR /opt/prefect/flows/
Deploy with:
from prefect import flow
from prefect.docker import DockerImage
@flow
def my_flow():
...
if __name__ == "__main__":
my_flow.deploy(
name="custom-dockerfile",
work_pool_name="my-docker-pool",
image=DockerImage(
name="my-registry/my-flows",
tag="v1",
dockerfile="Dockerfile",
),
push=True,
)
How to get your Docker worker running
- Create a Docker work pool:
prefect work-pool create "my-docker-pool" --type docker
- Start a worker that polls that pool:
prefect worker start --pool my-docker-pool
When to choose which option
- Pull from Git at runtime + EXTRA_PIP_PACKAGES:
- Frequent code changes, don’t want builds
- Simpler dependency sets
- Bake into image:
- Reproducible runs, faster startup
- Complex/native dependencies, private indexes, or air‑gapped
Helpful docs
- Docker worker/infrastructure: Run on Docker
- Storing flow code (Git, S3, etc.): Store flow code
A few clarifying questions to get you set up quickly:
- Is your Git repo public or private? If private, do you prefer passing a token or using a credentials block?
- Do you already have a requirements.txt/pyproject you want baked into an image, or would runtime installs via EXTRA_PIP_PACKAGES be fine?
- Do you have a Docker registry to push to if we go with the image build route?