Joseph Haaga

    Joseph Haaga

    1 year ago
    Getting this error when I try to use Docker storage
    ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
    Does
    python_dependencies
    take care of the
    pip install
    for me? Or do I have to provide a custom Dockerfile that does the
    COPY requirements.txt
    and
    RUN pip install -r requirements.txt
    ?
    Dylan

    Dylan

    1 year ago
    @Joseph Haaga
    python_dependencies
    should have taken care of that for you
    Can you post the full traceback?
    And your storage object, please
    Michael Adkins

    Michael Adkins

    1 year ago
    It’s also possible the dependency that you’re missing is your own module that the flow lives in.
    Joseph Haaga

    Joseph Haaga

    1 year ago
    print(f"Building Docker image '{flow.name}'")
        flow.storage = Docker(
            registry_url="<http://REDACTED.ecr.us-east-1.amazonaws.com|REDACTED.ecr.us-east-1.amazonaws.com>",
            image_name=flow.name,
            python_dependencies=reqs,
        )
    Most of the trace was
    pip install
    , but here is the rest
    Imported Flow: 'Extract Entities from CNBC'
    Building Docker image '{flow.name}'
    Setting Execution Environment
    Registering Flow 'Extract Entities from CNBC' to Project Relationship Discovery
    /Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/core/flow.py:1623: UserWarning: No result handler was specified on your Flow. Cloud features such as input caching and resuming task runs from failure may not work properly.
      registered_flow = client.register(
    
    [2020-10-30 21:08:08] INFO - prefect.Docker | Building the flow's Docker storage...
    Traceback (most recent call last):
      File "/Users/josephhaaga/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/Users/josephhaaga/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
        exec(code, run_globals)
      File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect_deployer/__main__.py", line 5, in <module>
        exit(main())
      File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect_deployer/main.py", line 27, in main
        print(f"Installing Flow requirements")
      File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/core/flow.py", line 1623, in register
        registered_flow = client.register(
      File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/client/client.py", line 734, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/core/flow.py", line 1458, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 351, in build
        self._build_image(push=push)
      File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 417, in _build_image
        raise ValueError(
    ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
    Dylan

    Dylan

    1 year ago
    @Joseph Haaga silly question, but are you sure you’ve specified all of your dependencies? Sometimes python deps require others (pandas requires psycopg2 and sqlalchemy if I use
    read_sql
    with postgres)
    Joseph Haaga

    Joseph Haaga

    1 year ago
    hmm I’ll double check but I believe I am; I froze my env to a
    requirements.txt
    and am parsing that to pass as the
    python_dependencies
    Michael Adkins

    Michael Adkins

    1 year ago
    It may also be worth looking at the discussion at https://prefect-community.slack.com/archives/CL09KU1K7/p1603476596033800
    Dylan

    Dylan

    1 year ago
    If you are, maybe double-check that the parsing is going as the storage object is expecting?
    Michael Adkins

    Michael Adkins

    1 year ago
    Although it's erroring in a different way so my apologies if that's a misleading direction.
    Joseph Haaga

    Joseph Haaga

    1 year ago
    Still getting this error on a very simple flow that only depends on
    prefect.Flow
    and
    prefect.task
    (the hello-world one). Is the typical deployment pattern to supply a base Dockerfile that COPY’s the source code into the image? I figured whatever
    _build_image
    occurs on
    flow.register
    would handle putting the source into the Docker image, but I’m becoming skeptical
    Michael Adkins

    Michael Adkins

    1 year ago
    Can you please share a full reproducible case with your code? (a redacted registry url is fine)