John O'Farrell
05/25/2022, 7:31 PMprefect.Docker:Building the flow's Docker storage...
after creating a new temp directory with the dockerfile and healthcheck.py
Kevin Kho
Kevin Kho
John O'Farrell
05/25/2022, 7:59 PMRUN pip install pip --upgrade
that is added to the end of it by prefect throws the error Could not install packages due to an OSError: [Errno 13] Permission denied: '/.venv/bin/pip'
. Could be related to some conflect between pip and pipenv, which I've been using to handle dependencies?Kevin Kho
John O'Farrell
05/25/2022, 8:08 PMENV PREFECT__USER_CONFIG_PATH='/opt/prefect/config.toml'
was a part of the dockerfile I've been using to build the project before I tried integrating it with prefect's docker storage. I'd really prefer to keep the compatibility with pipenv if possibleJohn O'Farrell
05/26/2022, 2:02 PMKevin Kho
John O'Farrell
05/26/2022, 3:20 PMCould not install packages due to an OSError: [Errno 13] Permission denied: '/.venv/bin/pip'
He's the portion of the image building where it fails
Step 3/8 : RUN pip install pip --upgrade
---> Running in 19aa18cb4c49
Requirement already satisfied: pip in /.venv/lib/python3.9/site-packages (22.0.4)
Collecting pip
Downloading pip-22.1.1-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 8.3 MB/s eta 0:00:00
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
Kevin Kho
DockerRun
or you do something like this with
flow.register(..., build=False)
and Prefect won’t try to build an image.John O'Farrell
05/26/2022, 3:38 PMKevin Kho
John O'Farrell
05/26/2022, 3:41 PMmarshmallow.exceptions.ValidationError: {'_schema': 'Invalid data type: None'}
John O'Farrell
05/26/2022, 3:42 PM.venv\lib\site-packages\prefect\agent\agent.py", line 388, in _deploy_flow_run
deployment_info = self.deploy_flow(flow_run
)Kevin Kho
John O'Farrell
05/26/2022, 3:48 PMflow.run_config = DockerRun(image="{image link}")
client = prefect.Client(api_key=os.environ['api key}])
id = client.register(flow=flow, project_name="{project name}", build=False)
launch_task_thread = threading.Thread(target=launch_task, args=(id,))
launch_task_thread.start()
DockerAgent().start()
The launch task thread waits 20 seconds before calling create_flow_run
with the newly registered id, allowing the flow to run after the Docker Agent is up and runningKevin Kho
John O'Farrell
05/26/2022, 3:52 PMKevin Kho
flow.storage = ...
just redact any sensitive info. or did you not define it anymore?John O'Farrell
05/26/2022, 3:57 PMJohn O'Farrell
05/26/2022, 3:58 PMValueError('Flow is not contained in this Storage')
Kevin Kho
John O'Farrell
05/26/2022, 4:36 PMFailed to load and execute flow run: FileNotFoundError(2, 'No such file or directory')
or a modulenotfound error depending on how I configure the path and prefect_directory parameters in docker to point to the file where the flow is writtenJohn O'Farrell
05/26/2022, 4:37 PM/home/appuser/src/app.py
Kevin Kho
John O'Farrell
05/26/2022, 4:42 PMroot:/home/appuser# ls
resources src
root:/home/appuser# cd src
root:/home/appuser/src# ls
__init__.py app.py transform transformation_tasks.py util.py validation
There's nothing stored in the opt
directory which prefect says it looks for by defaultKevin Kho
Docker(...)
looking for the pathJohn O'Farrell
05/26/2022, 4:55 PMdocker_storage = Docker(
image_name="ingestion-pipeline",
image_tag="prefect-test",
registry_url="{url}",
stored_as_script=True,
prefect_directory = "/home",
path=f"/home/appuser/src/app.py",
)
John O'Farrell
05/26/2022, 4:57 PMFailed to load and execute flow run: ModuleNotFoundError("No module named 'src'")
Kevin Kho
prefect_directory
but the Flow itself in app.py
actually started and has an import that is failing. Maybe you have something like
from src.transform import ...
but it can’t find src right? I have calls for the next hour, but basically this has to be installed as a Python module so it can be used in the container. Have you done that before?John O'Farrell
05/26/2022, 5:02 PMKevin Kho
setup.py
file to make the module pip installable. I have a walkthrough hereKevin Kho
John O'Farrell
05/26/2022, 9:44 PMFailed to load and execute flow run: ModuleNotFoundError("No module named 'src'")
if I try to pass in an environment variable dictionary into the run config. Confused why this would be the caseKevin Kho
John O'Farrell
05/26/2022, 9:51 PMrun_config=DockerRun(
image="{image}",
labels=["docker"],
),
it works up until it throws the key error for the missing env variableJohn O'Farrell
05/26/2022, 9:52 PMrun_config=DockerRun(
image="{image}",
env=env_dict,
labels=["docker"],
)
the it throws the module not found errorJohn O'Farrell
05/26/2022, 9:53 PMcreate_flow_run
parameters and when I change how it is originally definedKevin Kho
John O'Farrell
05/26/2022, 10:02 PMKevin Kho
John O'Farrell
05/26/2022, 10:13 PM