Fina Silva-Santisteban

    Fina Silva-Santisteban

    1 year ago
    Hi Prefect community, I hope everyone’s doing well! I need help setting up Docker Storage and using the docker prefect agent. My Dockerfile looks like this:
    FROM python:3.7
    WORKDIR /app
    COPY requirements.txt .
    RUN pip install -r requirements.txt
    COPY . .
    My flow like this:
    (...)
    storage = Docker(dockerfile="../Dockerfile")
    
    with Flow("Generate Report", result=LocalResult(), storage=storage) as flow:
    (...)
    When I try to register the flow, it seems to have issues with finding the
    requirements.txt
    file needed to create the image:
    [2021-01-25 15:21:03-0800] INFO - prefect.Docker | Building the flow's Docker storage...
    Step 1/14 : FROM python:3.7
     ---> ca194d6afe58
    Step 2/14 : WORKDIR /app
     ---> Using cache
     ---> a729913b700e
    Step 3/14 : COPY requirements.txt .
    COPY failed: stat /var/lib/docker/tmp/docker-builder883681743/requirements.txt: no such file or directory
    
    Traceback (most recent call last):
      File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 53, in <module>
        register_locally()
      File "/Users/fina/Documents/github/concierge_iro_reporting/etl/register_all_flows.py", line 21, in register_locally
        report_lt_target.register(project_name=project_name)
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1675, in register
        idempotency_key=idempotency_key,
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/client/client.py", line 783, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/core/flow.py", line 1450, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 363, in build
        self._build_image(push=push)
      File "/Users/fina/.pyenv/versions/3.7.3/lib/python3.7/site-packages/prefect/storage/docker.py", line 431, in _build_image
        "Your docker image failed to build!  Your flow might have "
    ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
    My folder is structure is like this:
    App
    -->prefect_flows
      -->my_flow.py
    requirements.txt
    Dockerfile
    Why is it looking for the requirements.txt inside
    /var/lib/docker/
    instead of the App’s directory? What am I missing? (The container runs normally when I use docker-compose to build and run the app)
    UPDATE: I’ve removed the lines in the Dockerfile
    COPY requirements.txt .
    RUN pip install -r requirements.txt
    and instead installed the necessary packages using the
    python_dependencies
    argument. So my updated flow file looks like this:
    (...)
    python_dependencies = ["pandas", "numpy",...]
    storage = Docker(dockerfile="../Dockerfile", python_dependencies=python_dependencies)
    
    with Flow("Generate Report", result=LocalResult(), storage=storage) as flow:
    (...)
    That seems to fix the issue with installing the necessary packages. But the next issue seems to be related to the folders inside the main app folder:
    Beginning health checks...
    System Version check: OK
    /opt/prefect/healthcheck.py:151: UserWarning: Flow uses module which is not importable. Refer to documentation on how to import custom modules <https://docs.prefect.io/api/latest/storage.html#docker>
      flows = cloudpickle_deserialization_check(flow_file_paths)
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 151, in <module>
        flows = cloudpickle_deserialization_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
        flows.append(cloudpickle.loads(flow_bytes))
    ModuleNotFoundError: No module named '(NAME OF ANY SUBFOLDER)'
    I have the impression that my Dockerfile is just not set up correctly. At the moment it only has:
    FROM python:3.7
    WORKDIR /app
    COPY . .
    What am I missing now?
    Kyle Moon-Wright

    Kyle Moon-Wright

    1 year ago
    Hey @Fina Silva-Santisteban, I think you’ll want to package in those files in your Storage
    env_vars
    kwarg with a
    PYTHONPATH
    argument. Another example can be found in the API docs.
    Fina Silva-Santisteban

    Fina Silva-Santisteban

    1 year ago
    Thank you, @Kyle Moon-Wright ! I didn’t realize that the folder containing the app’s logic would be considered
    other Python scripts
    . I’ll give that a try!
    Anze Kravanja

    Anze Kravanja

    1 year ago
    Adding to the Dockerfile:
    ENV PYTHONPATH "${PYTHONPATH}:."
    worked for me.