Jillian Kozyra
03/25/2021, 7:32 AMModuleNotFoundError: No module named 'data_flows'
when I try to register my flow locally using docker storage (specifically it’s failing the cloudpickle_deserialization_check
healthcheck). we’re using fully qualified imports in our flows (e.g. from data_flows.utils.data_extraction tasks import thing
) to satisfy mypy, so i’m hoping there’s a way to get this working as well. i think the config matches the tutorial, and i’ve tried different values for the python path without any luck. Here’s our Docker setup:
Docker(
registry_url=ecr_registry_url,
python_dependencies=python_dependencies,
files={
"/Users/jilliankozyra/projects/projectname/data_flows/utils/__init__.py": "/data_flows/utils/__init__.py",
"/Users/jilliankozyra/projects/projectname/data_flows/utils/install_dependencies.py": "/data_flows/utils/install_dependencies.py",
"/Users/jilliankozyra/projects/projectname/data_flows/utils/data_extraction_tasks.py": "/data_flows/utils/data_extraction_tasks.py"
},
env_vars={
"PYTHONPATH": "$PYTHONPATH:data_flows/"
},
)
Zanie
03/25/2021, 3:06 PM__init__.py
being copied in for the top-level data_flows
folderpip install
instead? It tends to be easier then setting a PYTHONPATH variable.Jillian Kozyra
03/25/2021, 6:22 PM/data_flows/_init_.py
but it does not appear to have made a difference.Zanie
03/25/2021, 6:40 PMsetup.py
file then pip install the local module.files
dict instead of using each fileJillian Kozyra
03/25/2021, 6:41 PMZanie
03/25/2021, 6:48 PMJillian Kozyra
03/25/2021, 6:48 PMexport PYTHONPATH=.:${PYTHONPATH}:/Users/jilliankozyra/projects/projectname/
data_flows
is a subdirectory of thatZanie
03/25/2021, 6:49 PM/
to your pythonpath in the docker image?Jillian Kozyra
03/25/2021, 6:51 PM