Mitchell Bregman
10/23/2020, 6:09 PMModuleNotFoundError: No module named src
during the flow healthcheck, traceback here. Am I doing something wrong in terms of __init__
packaging? This is a followup to thread yesterday.nicholas
src
is referenced, it's from within the src
directory, shouldn't the __init__
module reference it with from flow import Flow
?Mitchell Bregman
10/23/2020, 6:14 PMZanie
Mitchell Bregman
10/23/2020, 6:15 PMZanie
src
init file you should still reference the full path to the moduleMitchell Bregman
10/23/2020, 6:16 PM"""Top-level module."""
from src.flow import flow
__all__ = ["flow"]
flow.py
to build.py
Zanie
/opt/prefect/healthcheck.py:147: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/environments/storage.html#docker>
flows = cloudpickle_deserialization_check(flow_file_paths)
src
which is not installed within the docker containerMitchell Bregman
10/23/2020, 6:18 PMimport src
all works just fineZanie
pip install -e
is not run within the docker container thoughMitchell Bregman
10/23/2020, 6:19 PMnicholas
src
folder to the docker container?Zanie
extra_dockerfile_commands
kwarg or include your module like so
Docker(
files={
# absolute path source -> destination in image
"/Users/me/code/mod1.py": "/modules/mod1.py",
"/Users/me/code/mod2.py": "/modules/mod2.py",
},
env_vars={
# append modules directory to PYTHONPATH
"PYTHONPATH": "$PYTHONPATH:modules/"
},
)
Mitchell Bregman
10/23/2020, 6:22 PMZanie
"/path/in/ci/to/module": "/modules"
Mitchell Bregman
10/23/2020, 6:24 PMZanie
pip install -e /modules/yourmodule
(via the extra cmds) or add it to the PYTHONPATH using env_vars
Mitchell Bregman
10/23/2020, 6:38 PMextra_dockerfile_commands="pip install -e /modules",
files={f"{os.path.join(os.path.expanduser('~'), 'project')}": "/modules"},
flow.storage = Docker(
env_vars=config.ENVIRONMENT_VARIABLES,
extra_dockerfile_commands="pip install -e /modules",
files={f"{os.path.join(os.path.expanduser('~'), 'project')}": "/modules"},
image_name=config.DOCKER_IMAGE_NAME,
image_tag=config.DOCKER_IMAGE_TAG,
python_dependencies=config.PYTHON_DEPENDENCIES,
registry_url="<http://parkmobile-docker.jfrog.io|parkmobile-docker.jfrog.io>",
tls_config=tls_config,
)
os.path.join(os.path.expanduser('~'), 'project'
resolves to home/circleci/project
(all the code if you were to clone it lives here)/modules
Zanie
Mitchell Bregman
10/23/2020, 6:40 PMdocker.errors.APIError: 400 Client Error: Bad Request ("Dockerfile parse error line 16: unknown instruction: P")
Zanie
Mitchell Bregman
10/23/2020, 6:41 PMZanie
["RUN …"]
Mitchell Bregman
10/23/2020, 6:41 PMZanie
Miha Sajko
12/19/2020, 4:56 PMfiles
argument as discussed in this thread or is there a better way?Zanie
from my_project import PROJECT_PATH, PROJECT_NAME
from prefect.storage.docker import Docker
def ProjectDockerStorage(
project_path: str = PROJECT_PATH, project_name: str = PROJECT_NAME, **kwargs
) -> Docker:
"""
A thin wrapper around `prefect.storage.Docker` with installation of a local project,
defaulting to installing this project
Cannot be a class because then it is not a known serializable storage type so this
is just an instance factory for Docker storage
"""
# Copy this namespace into the docker image
kwargs.setdefault("files", {})
kwargs["files"][str(project_path)] = project_name
# Install the namespace so it's on the Python path
kwargs.setdefault("extra_dockerfile_commands", [])
kwargs["extra_dockerfile_commands"].append(f"RUN pip install -e {project_name}")
return Docker(**kwargs)
then
flow.storage = ProjectDockerStorage()
Sagun Garg
12/28/2020, 8:21 AM