https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> what is the best way to organize flow...
# ask-marvin
a
@Marvin what is the best way to organize flow code with uv
m
✅ thought for 135.6 seconds
A good pattern is to treat your Prefect code like a normal Python package, and let 
uv
manage the environment and dependencies. Recommended structure
Copy code
text
my-project/
├── pyproject.toml
├── uv.lock
├── prefect.yaml          # optional
├── .prefectignore
├── src/
│   └── my_project/
│       ├── flows/
│       │   ├── daily_etl.py
│       │   └── reporting.py
│       ├── tasks/
│       │   ├── extract.py
│       │   └── transform.py
│       ├── utils/
│       │   ├── config.py
│       │   └── db.py
│       └── __init__.py
└── tests/
Why this works well - 
uv
works cleanly with a standard 
pyproject.toml
- Prefect imports behave more predictably when your code is a package - deployments are easier to define with module-style entrypoints - shared logic stays out of flow files Suggested rules of thumb 1. Keep flows thin - flows should mostly orchestrate - business logic should live in tasks or plain helper functions 2. Put reusable units in 
tasks/
 - things you want retries, caching, logging, or mapping around 3. Put plain Python helpers in 
utils/
or domain modules - config loading - db clients - parsing / validation - shared constants 4. Use importable module paths - prefer entrypoints like 
my_project.flows.daily_etl:daily_etl
- avoid ad hoc scripts scattered around the repo Example
Copy code
text
src/my_project/flows/daily_etl.py
Copy code
python
from prefect import flow
from my_project.tasks.extract import fetch_data
from my_project.tasks.transform import clean_data

@flow
def daily_etl():
    raw = fetch_data()
    cleaned = clean_data(raw)
    return cleaned

if __name__ == "__main__":
    daily_etl()
Copy code
text
src/my_project/tasks/extract.py
Copy code
python
from prefect import task

@task(retries=3, retry_delay_seconds=5)
def fetch_data():
    return [{"x": 1}, {"x": 2}]
Copy code
text
src/my_project/tasks/transform.py
Copy code
python
from prefect import task

@task
def clean_data(rows):
    return [r for r in rows if "x" in r]
With 
uv
 - install deps with 
uv add prefect
- run flows with 
uv run python -m my_project.flows.daily_etl
- run tests with 
uv run pytest
That 
-m
style is nice because it uses package imports the same way your deployments will. 
pyproject.toml
basics
Copy code
toml
[project]
name = "my-project"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
  "prefect>=3",
]
Deployment tip If you deploy from source, use package entrypoints, not relative imports or one-off script paths when possible. For example:
Copy code
python
flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="src/my_project/flows/daily_etl.py:daily_etl",
)
Or if your environment installs the package and resolves modules cleanly, organize around the same package path consistently in your code. Bottom line The best way to organize flow code with 
uv
is: - use a standard Python package layout - keep flows in 
src/<package>/flows/
- keep reusable tasks in 
src/<package>/tasks/
- keep non-Prefect logic in helpers/modules - run everything through 
uv run
- pin dependencies with 
uv.lock
If you want, I can also give you: 1. a minimal starter repo layout for Prefect + 
uv
, or 2. a recommended 
prefect.yaml
+ deploy setup for this structure.
a
@Marvin what is the best way to deploy with uv using a container with dependencies and getting flow code from github
m
✅ thought for 119.4 seconds
Yes — the best general pattern for this in Prefect 3.x is: Use a container image for the runtime and system/Python dependencies, and use 
flow.from_source(...)
to pull the flow code from GitHub at run time. That gives you: - a stable execution environment - code updates from GitHub without rebuilding the image every time - clean separation between environment and flow source Recommended pattern 1. Build a container image that contains: - Python - Prefect - 
uv
- any OS/system libraries you need - optionally: your Python dependencies preinstalled 2. Store flow code in GitHub with: - 
pyproject.toml
- 
uv.lock
- your flow files 3. Deploy with 
flow.from_source(...)
pointing at the GitHub repo 4. Choose one of these dependency strategies: - Best for speed/reliability: bake Python deps into the image - Best for flexibility: let Prefect auto-install dependencies from the pulled repo using 
uv
--- Best practice recommendation If your flows run often or need predictable startup times: Prefer a custom image with dependencies already installed, and pull only the flow code from GitHub. If you want simpler iteration and don't mind slower startup: Use a thinner base image with 
uv
installed and enable runtime dependency installation. --- Pattern A: Best overall for production Container has dependencies, code comes from GitHub This is usually the sweet spot. Dockerfile
Copy code
dockerfile
FROM prefecthq/prefect:3-latest

WORKDIR /opt/prefect

COPY pyproject.toml uv.lock ./

RUN uv sync --frozen

ENV PATH="/opt/prefect/.venv/bin:$PATH"
Then deploy from GitHub source:
Copy code
python
from prefect import flow

@flow
def my_flow():
    print("hello")

if __name__ == "__main__":
    my_flow.from_source(
        source="<https://github.com/my-org/my-repo.git>",
        entrypoint="src/my_project/flows/my_flow.py:my_flow",
    ).deploy(
        name="my-flow",
        work_pool_name="my-docker-pool",
        image="my-registry/my-prefect-runtime:latest",
        build=False,
    )
Why this is good - fast startup - deterministic dependency set - GitHub can still provide the latest flow code - fewer runtime surprises Important caveat Your image dependencies and your repo dependencies must stay aligned. If the GitHub code starts importing a package not present in the image, runs will fail. --- Pattern B: Thin image + install deps from GitHub repo at runtime with 
uv
 This is great when the code changes often.
Copy code
python
from prefect import flow

@flow
def my_flow():
    print("hello")

if __name__ == "__main__":
    my_flow.from_source(
        source="<https://github.com/my-org/my-repo.git>",
        entrypoint="src/my_project/flows/my_flow.py:my_flow",
    ).deploy(
        name="my-flow",
        work_pool_name="my-docker-pool",
        build=False,
        job_variables={
            "env": {
                "PREFECT_RUNNER_AUTO_INSTALL_DEPENDENCIES": "true"
            }
        },
    )
For this to work well: - 
uv
must be available in the container - the pulled repo must have a 
pyproject.toml
at the repo root - 
prefect
must be listed in 
[project].dependencies
- your needed packages should be in normal dependencies, not only dev groups Tradeoff - simpler image management - slower startup - more runtime dependency resolution --- Which entrypoint format should you use? When pulling from GitHub source, file-path entrypoints are usually the safest:
Copy code
python
entrypoint="src/my_project/flows/my_flow.py:my_flow"
This is generally more reliable than a pure module path when code is being cloned into a runtime directory. --- How to organize the repo
Copy code
text
my-repo/
├── pyproject.toml
├── uv.lock
└── src/
    └── my_project/
        ├── flows/
        │   └── my_flow.py
        ├── tasks/
        │   └── common.py
        └── utils/
            └── config.py
That works well whether deps are baked into the image or installed at runtime. --- My practical recommendation For most teams: Use a custom image with Prefect + uv + all Python dependencies preinstalled, and use GitHub only for flow code retrieval. That usually gives the best balance of: - reproducibility - performance - easy code updates Use runtime auto-install only if: - your dependencies change frequently - you want to avoid rebuilding images often - slower cold starts are acceptable --- CLI note The modern deployment CLI is 
prefect deploy
in Prefect 3.x, not the old 2.x deployment build flow. Docs and references: - Prefect docs - Deployment docs - Work pools - Store flow code If you want, I can give you a complete working example of either: 1. Docker work pool + GitHub source + baked dependencies, or 2. Docker work pool + GitHub source + runtime 
uv
install
Open in Slack
PreviousNext
Powered by