Prefect 2.8.4: How do I structure my flow definiti...
# prefect-community
f
Prefect 2.8.4: How do I structure my flow definition code to enable code reuse?
I have the following project structure:
Copy code
.
├── flow_definitions
│   ├── __init__.py
│   ├── masterdata
│   │   ├── deployment.py
│   │   ├── extract_data.py
│   │   └── __init__.py
│   ├── classification
│   │   ├── deployment.py
│   │   ├── __init__.py
│   │   └── preprocessing.py
│   └── utils
│       ├── date_utils.py
│       ├── __init__.py
├── README.md
├── requirements.txt
└── setup.py
In
classification/preprocessing.py
, I am doing a sibling import as follows:
Copy code
from flows.flow_definitions.utils.date_utils import parse_date
However, this fails:
Copy code
ModuleNotFoundError: No module named 'flows'
since it seemingly can't find my parent module (
.
). Strictly speaking, this is a Python problem and not a prefect issue. However, I can't seem to find an example detailing this kind of approach. I thought that maybe the Deployment
path
parameter might have something to do with this scenario, but it seems that's not the case. So, is there an example of a project using common functionality? Do I have to install my
utils
module, or is there another preferred way to get this to work in my flows?
d
As you identified, it is a Python problem. Are you running any of the listed modules, or are they purely for import? If purely import, access with relatives.
Copy code
# classification/preprocessing.py
from ..utils.date_utils import parse_date
If both a module and script entrypoint
Copy code
# classification/preprocessing.py
try:
    from ..utils.date_utils import parse_date
except ImportError:
    from flows.flow_definitions.utils.date_utils import parse_date
Actually, the script entrypoint code may be off, can't remember how I used to do it. Typically you want them to be purely import, so try to make that work.