<@ULVA73B9P> What is the ideal file structure for ...
# ask-marvin
k
@Marvin What is the ideal file structure for a GitHub repository containing flows, deployments, etc.?
m
thought for 200.9 seconds
Great question. In Prefect 3.x, an “ideal” repo keeps flows, deployment config, infra setup, and CI clearly separated while remaining simple to navigate. Here’s a solid, small-to-medium project layout:
Copy code
my-prefect-project/
├── flows/                      # Your flows (group by domain or pipeline)
│   ├── hello.py
│   └── etl/
│       ├── extract.py
│       └── transform.py
├── tasks/                      # Reusable task modules (optional)
│   └── io.py
├── deployments/                # Python-based deployment scripts (optional)
│   └── deploy.py               # uses flow.from_source(...).deploy(...)
├── infrastructure/             # IaC-like setup for Prefect resources
│   ├── setup_blocks.py         # create/load Blocks programmatically
│   └── setup_work_pools.py
├── tests/                      # Unit/integration tests
│   └── test_flows.py
├── prefect.yaml                # Versioned deployment definitions (recommended)
├── requirements.txt or pyproject.toml
├── .prefectignore              # Exclude large/unneeded files from packaging
├── Dockerfile                  # Optional: custom image for workers/runs
└── .github/workflows/
    └── deploy.yml              # CI to run `prefect deploy`
If you prefer a Python “src” layout:
Copy code
my-prefect-project/
├── src/
│   └── my_project/
│       ├── flows/
│       ├── tasks/
│       └── utils/
├── deployments/
├── tests/
├── prefect.yaml
├── pyproject.toml
└── .github/workflows/deploy.yml
Monorepo pattern (one
prefect.yaml
per subproject):
Copy code
monorepo/
├── project_a/
│   ├── flows/
│   ├── prefect.yaml
│   └── pyproject.toml
├── project_b/
│   ├── flows/
│   ├── prefect.yaml
│   └── pyproject.toml
└── shared/
    └── lib/
Minimal examples - Minimal prefect.yaml (kept at repo root):
Copy code
name: my-project

pull:
  - prefect.deployments.steps.git_clone:
      repository: <https://github.com/org/repo.git>
      branch: main

deployments:
  - name: hello
    entrypoint: flows/hello.py:hello_flow
    work_pool:
      name: my-work-pool
- Python deployment script (optional, if you prefer code over YAML-only):
Copy code
# deployments/deploy.py
from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source="<https://github.com/org/repo.git>",
        entrypoint="flows/hello.py:hello_flow",
    ).deploy(
        name="hello",
        work_pool_name="my-work-pool",
    )
- CI to deploy on push (GitHub Actions):
Copy code
# .github/workflows/deploy.yml
name: Deploy Prefect

on:
  push:
    branches: [ main ]
    paths:
      - 'flows/**'
      - 'prefect.yaml'
      - 'pyproject.toml'
      - 'requirements.txt'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - env:
          PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
          PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
        run: prefect deploy --all
Practical tips - Keep flows importable and free of side effects at import time. - Use
.prefectignore
to exclude
tests/
, large data, and other non-runtime files from packaging. - Store credentials in Prefect Blocks and keep a small setup script (e.g.,
infrastructure/setup_blocks.py
) to create/update them. - Use work pools (and optionally queues) to separate environments (dev/stg/prod). You can keep one
prefect.yaml
with multiple deployments or multiple
prefect.*.yaml
files per environment. - Prefer
prefect deploy
(CLI) and/or
flow.from_source(...).deploy(...)
for creating/updating deployments in Prefect 3.x. If you share a bit about your target runtime (Process/Docker/Kubernetes), whether you’re in a monorepo, and how you’d like to deploy (YAML vs Python script), I can tailor this structure and give you a ready-to-use
prefect.yaml
and CI template.
k
@Marvin What if the repository contains other non-Prefect related code?
m
thought for 119.6 seconds
Totally fine. In Prefect 3.x you can keep your Prefect project alongside other app code. Here are practical patterns and tips to avoid pulling the entire repo into every deployment: Options for mixed repos - Keep a Prefect subproject: Place your Prefect files in a subdirectory (e.g., services/data-service/) with its own prefect.yaml and .prefectignore. - Use a custom prefect.yaml path: You can put prefect.yaml anywhere and target it with: -
prefect deploy --prefect-file services/data-service/prefect.yaml
- Use sparse checkout with git_clone to pull only needed directories: - In prefect.yaml:
Copy code
pull:
    - prefect.deployments.steps.git_clone:
        repository: <https://github.com/org/monorepo.git>
        branch: main
        directories: ["services/data-service", "shared"]
- Notes: - Supported fields include: repository, branch, commit_sha, directories, access_token, include_submodules, and credentials blocks (e.g., GitHubCredentials). - Not supported: depth (Prefect uses a shallow clone internally), repo_subdirectory (use directories instead). Entrypoints in subprojects - Entrypoints in deployments must be relative to the project root you’re deploying from, not the prefect.yaml file’s directory. This trips people up in monorepos. - Examples: - From repo root: -
prefect deploy --prefect-file services/data-service/prefect.yaml
- In the yaml:
entrypoint: services/data-service/flows/ingest.py:ingest_flow
- From inside services/data-service: -
prefect deploy --prefect-file prefect.yaml
- In the yaml:
entrypoint: flows/ingest.py:ingest_flow
Use .prefectignore to keep packaging small - Prefect uses .prefectignore (gitignore syntax) when building the deployment package. - Place one at your Prefect subproject root to exclude non-Prefect code:
Copy code
# Ignore build/test and other services
  __pycache__/
  *.pyc
  .pytest_cache/
  dist/
  build/
  *.egg-info/
  node_modules/
  .env
  # Ignore other services in the monorepo
  ../services/*/node_modules/
  ../services/*/dist/
  # Keep only what we need
  !flows/
  !shared/
- This keeps uploads fast and images small if you’re baking code into images. Recommended structure inside a mixed repo
Copy code
my-monorepo/
├── app/                        # Non-Prefect app code
├── services/
│   ├── data-service/
│   │   ├── flows/
│   │   ├── infrastructure/
│   │   │   ├── setup_blocks.py
│   │   │   └── setup_work_pools.py
│   │   ├── prefect.yaml
│   │   └── .prefectignore
│   └── analytics-service/
│       ├── flows/
│       ├── prefect.yaml
│       └── .prefectignore
└── shared/
    └── utils/
CLI you’ll use most - Deploy all deployments in a subproject:
Copy code
prefect deploy --prefect-file services/data-service/prefect.yaml --all
- Deploy just one deployment by name:
Copy code
prefect deploy --prefect-file services/data-service/prefect.yaml --name ingest-deployment
A couple more gotchas - If you commit multiple prefect.yaml files across services, make sure each uses pull: git_clone with directories to avoid copying the whole monorepo at runtime. - If you’re using flow.from_source(...).deploy(...) in a Python script, point source at the repo and keep entrypoint consistent with the paths above. Example:
Copy code
from prefect import flow

  if __name__ == "__main__":
      flow.from_source(
          source="<https://github.com/org/monorepo.git>",
          entrypoint="services/data-service/flows/ingest.py:ingest_flow",
      ).deploy(
          name="ingest",
          work_pool_name="my-work-pool",
      )
If you share your repo layout (paths and which parts are Prefect), I can draft a ready-to-use prefect.yaml and .prefectignore that only pulls what you need.