Kaleb King
12/16/2025, 7:38 PMMarvin
12/16/2025, 7:38 PMMarvin
12/16/2025, 7:42 PMmy-prefect-project/
├── flows/ # Your flows (group by domain or pipeline)
│ ├── hello.py
│ └── etl/
│ ├── extract.py
│ └── transform.py
├── tasks/ # Reusable task modules (optional)
│ └── io.py
├── deployments/ # Python-based deployment scripts (optional)
│ └── deploy.py # uses flow.from_source(...).deploy(...)
├── infrastructure/ # IaC-like setup for Prefect resources
│ ├── setup_blocks.py # create/load Blocks programmatically
│ └── setup_work_pools.py
├── tests/ # Unit/integration tests
│ └── test_flows.py
├── prefect.yaml # Versioned deployment definitions (recommended)
├── requirements.txt or pyproject.toml
├── .prefectignore # Exclude large/unneeded files from packaging
├── Dockerfile # Optional: custom image for workers/runs
└── .github/workflows/
└── deploy.yml # CI to run `prefect deploy`
If you prefer a Python “src” layout:
my-prefect-project/
├── src/
│ └── my_project/
│ ├── flows/
│ ├── tasks/
│ └── utils/
├── deployments/
├── tests/
├── prefect.yaml
├── pyproject.toml
└── .github/workflows/deploy.yml
Monorepo pattern (one prefect.yaml per subproject):
monorepo/
├── project_a/
│ ├── flows/
│ ├── prefect.yaml
│ └── pyproject.toml
├── project_b/
│ ├── flows/
│ ├── prefect.yaml
│ └── pyproject.toml
└── shared/
└── lib/
Minimal examples
- Minimal prefect.yaml (kept at repo root):
name: my-project
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/org/repo.git>
branch: main
deployments:
- name: hello
entrypoint: flows/hello.py:hello_flow
work_pool:
name: my-work-pool
- Python deployment script (optional, if you prefer code over YAML-only):
# deployments/deploy.py
from prefect import flow
if __name__ == "__main__":
flow.from_source(
source="<https://github.com/org/repo.git>",
entrypoint="flows/hello.py:hello_flow",
).deploy(
name="hello",
work_pool_name="my-work-pool",
)
- CI to deploy on push (GitHub Actions):
# .github/workflows/deploy.yml
name: Deploy Prefect
on:
push:
branches: [ main ]
paths:
- 'flows/**'
- 'prefect.yaml'
- 'pyproject.toml'
- 'requirements.txt'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- env:
PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
run: prefect deploy --all
Practical tips
- Keep flows importable and free of side effects at import time.
- Use .prefectignore to exclude tests/, large data, and other non-runtime files from packaging.
- Store credentials in Prefect Blocks and keep a small setup script (e.g., infrastructure/setup_blocks.py) to create/update them.
- Use work pools (and optionally queues) to separate environments (dev/stg/prod). You can keep one prefect.yaml with multiple deployments or multiple prefect.*.yaml files per environment.
- Prefer prefect deploy (CLI) and/or flow.from_source(...).deploy(...) for creating/updating deployments in Prefect 3.x.
If you share a bit about your target runtime (Process/Docker/Kubernetes), whether you’re in a monorepo, and how you’d like to deploy (YAML vs Python script), I can tailor this structure and give you a ready-to-use prefect.yaml and CI template.Kaleb King
12/16/2025, 7:44 PMMarvin
12/16/2025, 7:44 PMMarvin
12/16/2025, 7:46 PMprefect deploy --prefect-file services/data-service/prefect.yaml
- Use sparse checkout with git_clone to pull only needed directories:
- In prefect.yaml:
pull:
- prefect.deployments.steps.git_clone:
repository: <https://github.com/org/monorepo.git>
branch: main
directories: ["services/data-service", "shared"]
- Notes:
- Supported fields include: repository, branch, commit_sha, directories, access_token, include_submodules, and credentials blocks (e.g., GitHubCredentials).
- Not supported: depth (Prefect uses a shallow clone internally), repo_subdirectory (use directories instead).
Entrypoints in subprojects
- Entrypoints in deployments must be relative to the project root you’re deploying from, not the prefect.yaml file’s directory. This trips people up in monorepos.
- Examples:
- From repo root:
- prefect deploy --prefect-file services/data-service/prefect.yaml
- In the yaml: entrypoint: services/data-service/flows/ingest.py:ingest_flow
- From inside services/data-service:
- prefect deploy --prefect-file prefect.yaml
- In the yaml: entrypoint: flows/ingest.py:ingest_flow
Use .prefectignore to keep packaging small
- Prefect uses .prefectignore (gitignore syntax) when building the deployment package.
- Place one at your Prefect subproject root to exclude non-Prefect code:
# Ignore build/test and other services
__pycache__/
*.pyc
.pytest_cache/
dist/
build/
*.egg-info/
node_modules/
.env
# Ignore other services in the monorepo
../services/*/node_modules/
../services/*/dist/
# Keep only what we need
!flows/
!shared/
- This keeps uploads fast and images small if you’re baking code into images.
Recommended structure inside a mixed repo
my-monorepo/
├── app/ # Non-Prefect app code
├── services/
│ ├── data-service/
│ │ ├── flows/
│ │ ├── infrastructure/
│ │ │ ├── setup_blocks.py
│ │ │ └── setup_work_pools.py
│ │ ├── prefect.yaml
│ │ └── .prefectignore
│ └── analytics-service/
│ ├── flows/
│ ├── prefect.yaml
│ └── .prefectignore
└── shared/
└── utils/
CLI you’ll use most
- Deploy all deployments in a subproject:
prefect deploy --prefect-file services/data-service/prefect.yaml --all
- Deploy just one deployment by name:
prefect deploy --prefect-file services/data-service/prefect.yaml --name ingest-deployment
A couple more gotchas
- If you commit multiple prefect.yaml files across services, make sure each uses pull: git_clone with directories to avoid copying the whole monorepo at runtime.
- If you’re using flow.from_source(...).deploy(...) in a Python script, point source at the repo and keep entrypoint consistent with the paths above. Example:
from prefect import flow
if __name__ == "__main__":
flow.from_source(
source="<https://github.com/org/monorepo.git>",
entrypoint="services/data-service/flows/ingest.py:ingest_flow",
).deploy(
name="ingest",
work_pool_name="my-work-pool",
)
If you share your repo layout (paths and which parts are Prefect), I can draft a ready-to-use prefect.yaml and .prefectignore that only pulls what you need.