Luis Barragan
07/14/2026, 4:20 PMMarvin
07/14/2026, 4:21 PMMarvin
07/14/2026, 4:22 PMflows/sync.py for both related flows
- flows/common.py or flows/utils.py for shared tasks/helpers
Example:
python
from prefect import flow
@flow
def backfill_flow(start_date: str, end_date: str):
...
@flow
def recent_sync_flow():
...
This works well when:
- both flows operate on the same dataset/domain
- they reuse the same tasks/helpers
- they’re deployed together or from the same repo/image
When to split into separate files
Use separate files if:
- the flows are getting large or hard to navigate
- they use different infrastructure or work pools
- they have very different ownership or release cadence
- one is “operational” sync logic and the other is “historical repair” logic with lots of custom behavior
Example structure:
text
flows/
backfill.py
recent_sync.py
common.py
Practical rule of thumb
- Same file for a small number of closely related flows
- Separate files once readability starts to suffer
So for one backfill flow + one recent-sync flow, I’d usually start with the same file if they’re closely related. If the file grows or the behaviors diverge, split them later.
A couple relevant Prefect docs:
- Deploy via Python
- prefect.yaml
- Flows
If you want, I can also suggest a recommended project layout for a data pipeline with backfill + incremental sync + shared tasks.Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by