Kaleb King
07/05/2024, 12:50 PMMarvin
07/05/2024, 12:51 PMproject-root/
├── flows/
│ ├── flow1.py
│ ├── flow2.py
│ ├── ...
├── deployments/
│ ├── deployment1.yaml
│ ├── deployment2.yaml
│ ├── ...
└── prefect.yaml
### Key Components:
1. flows/: This directory contains the definitions of your flows. Each flow can be in its own Python file (e.g., flow1.py
for the first flow, flow2.py
for the second, etc.).
2. deployments/: This directory includes deployment configuration files. Each deployment configuration is defined in a separate YAML file (e.g., deployment1.yaml
, deployment2.yaml
, etc.). These configurations specify parameters such as entry points, schedules, work queues, and other deployment-specific settings.
3. prefect.yaml: This is the main configuration file for your Prefect project. It includes metadata and common settings that might be shared across deployments.
### Example `prefect.yaml`:
Inside the prefect.yaml
file, you might have:
yaml
name: my_prefect_project
prefect-version: 2.x
defintions:
work_pools:
default_pool:
name: my-work-pool
work_queue_name: my-work-queue
deployments:
- path: deployments/deployment1.yaml
- path: deployments/deployment2.yaml
### Example Deployment Configuration (deployment1.yaml
):
A deployment configuration could look like this:
yaml
name: Deployment 1
flow_name: flow1
entrypoint: flows/flow1.py:main
parameters:
key1: value1
key2: value2
work_pool:
name: my-work-pool
work_queue_name: my-work-queue
schedule:
interval: "@daily"
### Deployment Workflow:
To deploy the flows, you would use the prefect deploy
command against your deployment files:
bash
prefect deploy --deployments deployments/deployment1.yaml
prefect deploy --deployments deployments/deployment2.yaml
This layout ensures that each deployment is clearly separated and independently manageable.
For more details, you may refer to:
- Allow deployment name to be used as a folder in S3: https://github.com/PrefectHQ/prefect/issues/10362
- Prompt for deployment names when running `prefect deploy`: https://github.com/PrefectHQ/prefect/issues/9908Nate
07/05/2024, 12:57 PMprefect.yaml
at root of the repo, and then whatever other project structure makes sense
its technically possible to keep that file elsewhere, but imo its easier to think about at root