Hi Prefect community I hope everyone s doing well I need hel Prefect Community #ask-community

Hi Prefect community, I hope everyone’s doing well...

Fina Silva-Santisteban

01/25/2021, 11:36 PM

Hi Prefect community, I hope everyone’s doing well! I need help setting up Docker Storage and using the docker prefect agent. My Dockerfile looks like this:

Copy code

FROM python:3.7
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

My flow like this:

Copy code

(...)
storage = Docker(dockerfile="../Dockerfile")

with Flow("Generate Report", result=LocalResult(), storage=storage) as flow:
(...)

When I try to register the flow, it seems to have issues with finding the

requirements.txt

file needed to create the image:

Copy code

[2021-01-25 15:21:03-0800] INFO - prefect.Docker | Building the flow's Docker storage...
Step 1/14 : FROM python:3.7
 ---> ca194d6afe58
Step 2/14 : WORKDIR /app
 ---> Using cache
 ---> a729913b700e
Step 3/14 : COPY requirements.txt .
COPY failed: stat /var/lib/docker/tmp/docker-builder883681743/requirements.txt: no such file or directory

Traceback (most recent call last):
  File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 53, in <module>
    register_locally()
  File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 21, in register_locally
    report_lt_target.register(project_name=project_name)
  File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1675, in register
    idempotency_key=idempotency_key,
  File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 783, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1450, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 363, in build
    self._build_image(push=push)
  File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 431, in _build_image
    "Your docker image failed to build!  Your flow might have "
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.

My folder is structure is like this:

Copy code

App
-->prefect_flows
  -->my_flow.py
requirements.txt
Dockerfile

Why is it looking for the requirements.txt inside

/var/lib/docker/

instead of the App’s directory? What am I missing? (The container runs normally when I use docker-compose to build and run the app)

Fina Silva-Santisteban

01/26/2021, 12:22 AM

UPDATE: I’ve removed the lines in the Dockerfile

Copy code

COPY requirements.txt .
RUN pip install -r requirements.txt

and instead installed the necessary packages using the

python_dependencies

argument. So my updated flow file looks like this:

Copy code

(...)
python_dependencies = ["pandas", "numpy",...]
storage = Docker(dockerfile="../Dockerfile", python_dependencies=python_dependencies)

with Flow("Generate Report", result=LocalResult(), storage=storage) as flow:
(...)

That seems to fix the issue with installing the necessary packages. But the next issue seems to be related to the folders inside the main app folder:

Copy code

Beginning health checks...
System Version check: OK
/opt/prefect/healthcheck.py:151: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/storage.html#docker>
  flows = cloudpickle_deserialization_check(flow_file_paths)
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 151, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
ModuleNotFoundError: No module named '(NAME OF ANY SUBFOLDER)'

I have the impression that my Dockerfile is just not set up correctly. At the moment it only has:

Copy code

FROM python:3.7
WORKDIR /app
COPY . .

What am I missing now?

Kyle Moon-Wright

01/26/2021, 12:30 AM

Hey @Fina Silva-Santisteban, I think you’ll want to package in those files in your Storage

env_vars

kwarg with a

PYTHONPATH

argument. Another example can be found in the API docs.

🙌 1

Fina Silva-Santisteban

01/26/2021, 1:00 AM

Thank you, @Kyle Moon-Wright ! I didn’t realize that the folder containing the app’s logic would be considered

other Python scripts

. I’ll give that a try!

Anze Kravanja

01/30/2021, 12:41 AM

Adding to the Dockerfile:

Copy code

ENV PYTHONPATH "${PYTHONPATH}:."

worked for me.

🙏 1

Open in Slack

Previous Next