https://prefect.io logo
j

Joseph Haaga

10/30/2020, 9:11 PM
Getting this error when I try to use Docker storage
Copy code
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
Does
python_dependencies
take care of the
pip install
for me? Or do I have to provide a custom Dockerfile that does the
COPY requirements.txt
and
RUN pip install -r requirements.txt
?
d

Dylan

10/30/2020, 9:18 PM
@Joseph Haaga
python_dependencies
should have taken care of that for you
Can you post the full traceback?
And your storage object, please
z

Zanie

10/30/2020, 9:19 PM
It’s also possible the dependency that you’re missing is your own module that the flow lives in.
j

Joseph Haaga

10/30/2020, 9:20 PM
Copy code
print(f"Building Docker image '{flow.name}'")
    flow.storage = Docker(
        registry_url="<http://REDACTED.ecr.us-east-1.amazonaws.com|REDACTED.ecr.us-east-1.amazonaws.com>",
        image_name=flow.name,
        python_dependencies=reqs,
    )
Most of the trace was
pip install
, but here is the rest
Copy code
Imported Flow: 'Extract Entities from CNBC'
Building Docker image '{flow.name}'
Setting Execution Environment
Registering Flow 'Extract Entities from CNBC' to Project Relationship Discovery
/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/core/flow.py:1623: UserWarning: No result handler was specified on your Flow. Cloud features such as input caching and resuming task runs from failure may not work properly.
  registered_flow = client.register(

[2020-10-30 21:08:08] INFO - prefect.Docker | Building the flow's Docker storage...
Traceback (most recent call last):
  File "/Users/josephhaaga/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/josephhaaga/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect_deployer/__main__.py", line 5, in <module>
    exit(main())
  File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect_deployer/main.py", line 27, in main
    print(f"Installing Flow requirements")
  File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/core/flow.py", line 1623, in register
    registered_flow = client.register(
  File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/client/client.py", line 734, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/core/flow.py", line 1458, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 351, in build
    self._build_image(push=push)
  File "/Users/josephhaaga/Documents/Code/ETL/prefect-deployer/venv/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 417, in _build_image
    raise ValueError(
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
d

Dylan

10/30/2020, 9:43 PM
@Joseph Haaga silly question, but are you sure you’ve specified all of your dependencies? Sometimes python deps require others (pandas requires psycopg2 and sqlalchemy if I use
read_sql
with postgres)
j

Joseph Haaga

10/30/2020, 9:45 PM
hmm I’ll double check but I believe I am; I froze my env to a
requirements.txt
and am parsing that to pass as the
python_dependencies
👍 1
z

Zanie

10/30/2020, 9:46 PM
It may also be worth looking at the discussion at https://prefect-community.slack.com/archives/CL09KU1K7/p1603476596033800
d

Dylan

10/30/2020, 9:46 PM
If you are, maybe double-check that the parsing is going as the storage object is expecting?
z

Zanie

10/30/2020, 9:48 PM
Although it's erroring in a different way so my apologies if that's a misleading direction.
j

Joseph Haaga

10/30/2020, 10:29 PM
Still getting this error on a very simple flow that only depends on
prefect.Flow
and
prefect.task
(the hello-world one). Is the typical deployment pattern to supply a base Dockerfile that COPY’s the source code into the image? I figured whatever
_build_image
occurs on
flow.register
would handle putting the source into the Docker image, but I’m becoming skeptical
z

Zanie

10/30/2020, 11:40 PM
Can you please share a full reproducible case with your code? (a redacted registry url is fine)