Hey i m getting `ModuleNotFoundError No module named data fl Prefect Community #prefect-server

Hey, i’m getting `ModuleNotFoundError: No module n...

Jillian Kozyra

03/25/2021, 7:32 AM

Hey, i’m getting

ModuleNotFoundError: No module named 'data_flows'

when I try to register my flow locally using docker storage (specifically it’s failing the

cloudpickle_deserialization_check

healthcheck). we’re using fully qualified imports in our flows (e.g.

from data_flows.utils.data_extraction tasks import thing

) to satisfy mypy, so i’m hoping there’s a way to get this working as well. i think the config matches the tutorial, and i’ve tried different values for the python path without any luck. Here’s our Docker setup:

Copy code

Docker(
    registry_url=ecr_registry_url,
    python_dependencies=python_dependencies,
    files={
        "/Users/jilliankozyra/projects/projectname/data_flows/utils/__init__.py": "/data_flows/utils/__init__.py",
        "/Users/jilliankozyra/projects/projectname/data_flows/utils/install_dependencies.py": "/data_flows/utils/install_dependencies.py",
        "/Users/jilliankozyra/projects/projectname/data_flows/utils/data_extraction_tasks.py": "/data_flows/utils/data_extraction_tasks.py"
    },
    env_vars={
        "PYTHONPATH": "$PYTHONPATH:data_flows/"
    },
)

Zanie

03/25/2021, 3:06 PM

This likely isn't working because there's not an

__init__.py

being copied in for the top-level

data_flows

folder

Zanie

03/25/2021, 3:07 PM

Have you considered just using

pip install

instead? It tends to be easier then setting a PYTHONPATH variable.

Jillian Kozyra

03/25/2021, 6:22 PM

these are not pip-installable dependences. i’ve explicitly added a

/data_flows/_init_.py

but it does not appear to have made a difference.

Zanie

03/25/2021, 6:40 PM

You can just add a

setup.py

file then pip install the local module.

Zanie

03/25/2021, 6:41 PM

Also note you can copy a whole directory with the

files

dict instead of using each file

Zanie

03/25/2021, 6:41 PM

Have you tested the import locally instead of inside your docker image? It'll be faster to iterate that way.

Jillian Kozyra

03/25/2021, 6:41 PM

yes, it works fine locally, it just fails in the docker build

Jillian Kozyra

03/25/2021, 6:43 PM

i’m only explicitly copying files for testing purposes - the original code that handles this (not written by me) copies everything in every directory, which makes iterating slow

Zanie

03/25/2021, 6:48 PM

What's your local PYTHONPATH look like?

Jillian Kozyra

03/25/2021, 6:48 PM

export PYTHONPATH=.:${PYTHONPATH}:/Users/jilliankozyra/projects/projectname/

Jillian Kozyra

03/25/2021, 6:49 PM

data_flows

is a subdirectory of that

Zanie

03/25/2021, 6:49 PM

Ah. have you tried adding just

to your pythonpath in the docker image?

Jillian Kozyra

03/25/2021, 6:51 PM

i think that’s fixed it! thanks!

Open in Slack

Previous Next