<@ULVA73B9P> What is the best practice regarding t...
# ask-marvin
l
@Marvin What is the best practice regarding the location of flows? Let's say I have multiple flows, one for backfill and one for syncing recent data. Should I have them in the same file or in separate Python files?
m
thought for 108.6 seconds
Short answer: either is fine in Prefect 3.x. The best practice is to organize flows by shared purpose and lifecycle, not by a strict “one flow per file” rule. For your example: - If backfill and recent sync share logic, dependencies, and infrastructure, keeping them in the same file is totally reasonable. - If they have different schedules, teams, infra needs, or complexity, separate files is usually cleaner. What I’d recommend for your case A common pattern is: -
flows/sync.py
for both related flows -
flows/common.py
or
flows/utils.py
for shared tasks/helpers Example:
Copy code
python
from prefect import flow

@flow
def backfill_flow(start_date: str, end_date: str):
    ...

@flow
def recent_sync_flow():
    ...
This works well when: - both flows operate on the same dataset/domain - they reuse the same tasks/helpers - they’re deployed together or from the same repo/image When to split into separate files Use separate files if: - the flows are getting large or hard to navigate - they use different infrastructure or work pools - they have very different ownership or release cadence - one is “operational” sync logic and the other is “historical repair” logic with lots of custom behavior Example structure:
Copy code
text
flows/
  backfill.py
  recent_sync.py
  common.py
Practical rule of thumb - Same file for a small number of closely related flows - Separate files once readability starts to suffer So for one backfill flow + one recent-sync flow, I’d usually start with the same file if they’re closely related. If the file grows or the behaviors diverge, split them later. A couple relevant Prefect docs: - Deploy via Python - prefect.yaml - Flows If you want, I can also suggest a recommended project layout for a data pipeline with backfill + incremental sync + shared tasks.