Cullen Baker
08/28/2025, 3:34 PMMarvin
08/28/2025, 3:34 PMMarvin
08/28/2025, 3:48 PM# prefect-repo/flows/my_flow.py
from prefect import flow, task
from xlib.transform import do_thing # from repo X
@task
def run():
return do_thing()
@flow
def my_flow():
return run()
if __name__ == "__main__":
my_flow()
2) Make repo X available at runtime
Pick one of these options (all keep Prefect out of repo X):
- Package via Git URL
- Bake into your runtime image:
FROM prefecthq/prefect:3-latest
RUN pip install "git+<https://github.com/your-org/repo-x@<commit-or-tag>>" \
&& pip install pandas # any other deps
- Pin to a tag/commit for reproducibility.
- Internal package index / artifact
- Build a wheel for repo X (e.g., pip wheel
) and host it on your internal PyPI or artifact store; pip install it in the image/environment.
- Vendor/submodule
- Add repo X as a git submodule in the Prefect repo and adjust PYTHONPATH
so import xlib
works. This keeps repos separate but avoids extra packaging infra.
For private Git, ensure your worker environment can authenticate (e.g., PAT via standard Git credential helpers or environment variables), or bake the dependency into the image to avoid runtime fetches.
3) Deploy flows from your Prefect repo
Use Flow.from_source(...).deploy(...)
to create deployments that pull code from your Prefect repo:
from prefect.flows import Flow
deployment_flow = Flow.from_source(
source="<https://github.com/your-org/prefect-repo>",
entrypoint="flows/my_flow.py:my_flow",
)
deployment_flow.deploy(
name="my-flow-prod",
work_pool_name="your-pool",
# Point to an image that already has repo X installed
image="<http://ghcr.io/your-org/prefect-runtime:py3.11-xlib-1.2.3|ghcr.io/your-org/prefect-runtime:py3.11-xlib-1.2.3>",
cron="0 * * * *",
)
- Your worker should use the same image (Kubernetes/Docker workers) or a Python environment where repo X is installed (Process workers).
This keeps concerns separated:
- Repo X: business logic (no Prefect)
- Prefect repo: orchestration, deployments
- Worker/image: bundles dependencies
Notes and alternatives
- If you prefer YAML-based deployments for the Prefect repo, you can use prefect.yaml
to define deployments and an image that includes repo X; see prefect.yaml deployments.
- For what’s new in 3.x deployment flows/images, see Prefect 3.2 release notes.
A couple of quick questions to tailor this:
- Is repo X public or private? If private, do you prefer baking it into the image or authenticating at runtime?
- Which worker type are you using (Kubernetes, Docker, Process)?
- Do you have a container registry where we can publish a runtime image that includes repo X?