https://prefect.io logo
Powered by
Title
j

Jillian Kozyra

03/25/2021, 7:32 AM
Hey, i’m getting 
ModuleNotFoundError: No module named 'data_flows'
when I try to register my flow locally using docker storage (specifically it’s failing the 
cloudpickle_deserialization_check
healthcheck). we’re using fully qualified imports in our flows (e.g. 
from data_flows.utils.data_extraction tasks import thing
) to satisfy mypy, so i’m hoping there’s a way to get this working as well. i think the config matches the tutorial, and i’ve tried different values for the python path without any luck. Here’s our Docker setup: 
Docker(
    registry_url=ecr_registry_url,
    python_dependencies=python_dependencies,
    files={
        "/Users/jilliankozyra/projects/projectname/data_flows/utils/__init__.py": "/data_flows/utils/__init__.py",
        "/Users/jilliankozyra/projects/projectname/data_flows/utils/install_dependencies.py": "/data_flows/utils/install_dependencies.py",
        "/Users/jilliankozyra/projects/projectname/data_flows/utils/data_extraction_tasks.py": "/data_flows/utils/data_extraction_tasks.py"
    },
    env_vars={
        "PYTHONPATH": "$PYTHONPATH:data_flows/"
    },
)
z

Zanie

03/25/2021, 3:06 PM
This likely isn't working because there's not an 
__init__.py
being copied in for the top-level 
data_flows
folder
Have you considered just using 
pip install
instead? It tends to be easier then setting a PYTHONPATH variable.
j

Jillian Kozyra

03/25/2021, 6:22 PM
these are not pip-installable dependences. i’ve explicitly added a 
/data_flows/_init_.py
but it does not appear to have made a difference.
z

Zanie

03/25/2021, 6:40 PM
You can just add a 
setup.py
file then pip install the local module.
Also note you can copy a whole directory with the 
files
dict instead of using each file
Have you tested the import locally instead of inside your docker image? It'll be faster to iterate that way.
j

Jillian Kozyra

03/25/2021, 6:41 PM
yes, it works fine locally, it just fails in the docker build
i’m only explicitly copying files for testing purposes - the original code that handles this (not written by me) copies everything in every directory, which makes iterating slow
z

Zanie

03/25/2021, 6:48 PM
What's your local PYTHONPATH look like?
j

Jillian Kozyra

03/25/2021, 6:48 PM
export PYTHONPATH=.:${PYTHONPATH}:/Users/jilliankozyra/projects/projectname/
data_flows
is a subdirectory of that
z

Zanie

03/25/2021, 6:49 PM
Ah. have you tried adding just 
/
to your pythonpath in the docker image?
j

Jillian Kozyra

03/25/2021, 6:51 PM
i think that’s fixed it! thanks!
#prefect-serverJoin Slack
Powered by