<@ULVA73B9P> Let's say I have my prefect repo and ...
# ask-marvin
c
@Marvin Let's say I have my prefect repo and a separate repo X for another purpose. If I want to create a few flows using code from repo X, but I don't want to add prefect as a dependency in that repo, what is the best way to go about this?
m
thought for 822.7 seconds
Great question — you can keep Prefect completely out of repo X. The clean pattern in Prefect 3.x is: - Keep your flows in your Prefect repo (thin wrappers). - Treat repo X as a normal Python dependency that your runtime environment installs. - Deploy your flows from the Prefect repo; make sure the worker runtime (image or environment) has repo X installed. A simple, reliable setup 1) Author flows in the Prefect repo and import repo X - Your flow code just imports functions/classes from repo X; repo X does not import Prefect.
Copy code
# prefect-repo/flows/my_flow.py
from prefect import flow, task
from xlib.transform import do_thing  # from repo X

@task
def run():
    return do_thing()

@flow
def my_flow():
    return run()

if __name__ == "__main__":
    my_flow()
2) Make repo X available at runtime Pick one of these options (all keep Prefect out of repo X): - Package via Git URL - Bake into your runtime image:
Copy code
FROM prefecthq/prefect:3-latest
    RUN pip install "git+<https://github.com/your-org/repo-x@<commit-or-tag>>" \
        && pip install pandas  # any other deps
- Pin to a tag/commit for reproducibility. - Internal package index / artifact - Build a wheel for repo X (e.g.,
pip wheel
) and host it on your internal PyPI or artifact store; pip install it in the image/environment. - Vendor/submodule - Add repo X as a git submodule in the Prefect repo and adjust
PYTHONPATH
so
import xlib
works. This keeps repos separate but avoids extra packaging infra. For private Git, ensure your worker environment can authenticate (e.g., PAT via standard Git credential helpers or environment variables), or bake the dependency into the image to avoid runtime fetches. 3) Deploy flows from your Prefect repo Use
Flow.from_source(...).deploy(...)
to create deployments that pull code from your Prefect repo:
Copy code
from prefect.flows import Flow

deployment_flow = Flow.from_source(
    source="<https://github.com/your-org/prefect-repo>",
    entrypoint="flows/my_flow.py:my_flow",
)

deployment_flow.deploy(
    name="my-flow-prod",
    work_pool_name="your-pool",
    # Point to an image that already has repo X installed
    image="<http://ghcr.io/your-org/prefect-runtime:py3.11-xlib-1.2.3|ghcr.io/your-org/prefect-runtime:py3.11-xlib-1.2.3>",
    cron="0 * * * *",
)
- Your worker should use the same image (Kubernetes/Docker workers) or a Python environment where repo X is installed (Process workers). This keeps concerns separated: - Repo X: business logic (no Prefect) - Prefect repo: orchestration, deployments - Worker/image: bundles dependencies Notes and alternatives - If you prefer YAML-based deployments for the Prefect repo, you can use
prefect.yaml
to define deployments and an image that includes repo X; see prefect.yaml deployments. - For what’s new in 3.x deployment flows/images, see Prefect 3.2 release notes. A couple of quick questions to tailor this: - Is repo X public or private? If private, do you prefer baking it into the image or authenticating at runtime? - Which worker type are you using (Kubernetes, Docker, Process)? - Do you have a container registry where we can publish a runtime image that includes repo X?