Matthew Blau
01/27/2021, 9:01 PMValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
The full traceback is
Step 12/22 : WORKDIR /app
---> Running in b820b907ef61
Removing intermediate container b820b907ef61
---> fee83981b109
Step 13/22 : RUN groupadd --gid ${GID} ${USERNAME} && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME} && chown -R ${USERNAME} /app && chgrp -R ${USERNAME} /app
---> Running in 7f6c8b47c9e3
groupadd: invalid group ID 'rainbow'
Removing intermediate container 7f6c8b47c9e3
The command '/bin/sh -c groupadd --gid ${GID} ${USERNAME} && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME} && chown -R ${USERNAME} /app && chgrp -R ${USERNAME} /app' returned a non-zero code: 3
Traceback (most recent call last):
File "integration.py", line 11, in <module>
from change_password import *
File "/home/mblau/projects/experian-integration/change_password.py", line 5, in <module>
from integration import modify_config
File "/home/mblau/projects/experian-integration/integration.py", line 338, in <module>
flow.register(project_name="test")
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
registered_flow = client.register(
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/client/client.py", line 782, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1451, in serialize
storage = self.storage.build() # type: Optional[Storage]
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 360, in build
self._build_image(push=push)
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 427, in _build_image
raise ValueError(
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
what am I doing wrong?josh
01/27/2021, 9:20 PMfiles
kwarg and if there are extra pip-installable dependencies they can be specified with python_dependencies
Matthew Blau
01/27/2021, 9:23 PMjosh
01/27/2021, 9:29 PM/modules/
directory and then adds that directory to the PYTHONPATH so they can be imported. The dest is up to you but just not you need to add it to the PYTHONPATH so your flow can import themMatthew Blau
01/27/2021, 9:35 PMCOPY *.py /app/
COPY ./sql/* /app/sql/
WORKDIR /app
with Flow("example",
storage = Docker(dockerfile="/home/mblau/projects/experian-integration/Dockerfile",
files={'/home/mblau/projects/experian-integration/integration.py' : '/app',
'/home/mblau/projects/experian-integration/integration_lookups.py' : '/app',
'/home/mblau/projects/experian-integration/change_password.py' : '/app'})
) as flow:
josh
01/27/2021, 9:44 PM/app/change_password.py
. All that you need to do is set the python path env like the docs does to add it to the path so it can be imported:
env_vars={
# append modules directory to PYTHONPATH
"PYTHONPATH": "$PYTHONPATH:app/"
},
Matthew Blau
01/27/2021, 9:59 PMStep 11/23 : RUN groupadd --gid ${GID} ${USERNAME} && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME} && chown -R ${USERNAME} /app && chgrp -R ${USERNAME} /app
---> Running in db46d0aafd98
groupadd: invalid group ID 'rainbow'
Removing intermediate container db46d0aafd98
The command '/bin/sh -c groupadd --gid ${GID} ${USERNAME} && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME} && chown -R ${USERNAME} /app && chgrp -R ${USERNAME} /app' returned a non-zero code: 3
Traceback (most recent call last):
File "integration.py", line 11, in <module>
from change_password import *
File "/home/mblau/projects/experian-integration/change_password.py", line 5, in <module>
from integration import modify_config
File "/home/mblau/projects/experian-integration/integration.py", line 349, in <module>
flow.register(project_name="test")
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
registered_flow = client.register(
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/client/client.py", line 782, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1451, in serialize
storage = self.storage.build() # type: Optional[Storage]
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 360, in build
self._build_image(push=push)
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 427, in _build_image
raise ValueError(
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
Code has been adjusted to
with Flow("example",
storage = Docker(dockerfile="/home/mblau/projects/experian-integration/Dockerfile",
files={'/home/mblau/projects/experian-integration/integration.py' :
'experian-integration/integration.py/app',
'/home/mblau/projects/experian-integration/integration_lookups.py' :
'integration_lookups.py/app',
'/home/mblau/projects/experian-integration/change_password.py' :
'change_password.py/app'},
env_vars={
# append modules directory to PYTHONPATH
"PYTHONPATH": "$PYTHONPATH:app/"
},)
josh
01/27/2021, 10:05 PMfile.py/app
and not app/file.py
?Matthew Blau
01/27/2021, 10:06 PMjosh
01/27/2021, 10:20 PMfiles
attribute is used when not using a custom Dockerfile. Working with a dockerfile makes the assumption that your dockerfile copies over the appropriate files. In your Dockerfile maybe adding this will help:
COPY file.py /app/file.py
..for all your files..
ENV PYTHONPATH="${PYTHONPATH}:/app"
Instead of using env and files on the Docker storage objectMatthew Blau
01/27/2021, 10:29 PMStep 13/24 : ENV PYTHONPATH = "${PYTHONPATH"}:/app"
failed to process "= \"${PYTHONPATH\"}:/app\"": missing ':' in substitution
is the new errorRemoving intermediate container 9d8046a5b267
---> 6c27481596d3
Step 15/24 : RUN groupadd --gid ${GID} ${USERNAME} && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME} && chown -R ${USERNAME} /app && chgrp -R ${USERNAME} /app
---> Running in d08baee33e77
groupadd: invalid group ID 'rainbow'
Removing intermediate container d08baee33e77
The command '/bin/sh -c groupadd --gid ${GID} ${USERNAME} && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME} && chown -R ${USERNAME} /app && chgrp -R ${USERNAME} /app' returned a non-zero code: 3
docker-compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
and I am unsure of what Prefect is expecting for the dict. I know it's after 5pm as well on the east coast of the US so I hope I am not bothering you too much. I can always wait til tomorrow for an answer, but wanted to put my latest finds out there at least 🙂josh
01/27/2021, 10:54 PMDocker(build_kwargs={"UID": ..., "GID": ...})
Matthew Blau
01/28/2021, 1:29 PMStep 22/22 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/example.prefect"]' '(3, 8)'
---> Running in d892350e6f62
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 450, in _request
json_resp = response.json()
File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 147, in <module>
flows = cloudpickle_deserialization_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
flows.append(cloudpickle.load(f))
File "/app/integration.py", line 341, in <module>
flow.register(project_name="test")
File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
registered_flow = client.register(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 773, in register
project = self.graphql(query_project).data.project # type: ignore
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 297, in graphql
result = <http://self.post|self.post>(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 212, in post
response = self._request(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 453, in _request
raise ClientError(
prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
Removing intermediate container d892350e6f62
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/example.prefect"]' '(3, 8)'' returned a non-zero code: 1
Traceback (most recent call last):
File "integration.py", line 11, in <module>
from change_password import *
File "/home/mblau/projects/experian-integration/change_password.py", line 5, in <module>
from integration import *
File "/home/mblau/projects/experian-integration/integration.py", line 341, in <module>
flow.register(project_name="test")
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
registered_flow = client.register(
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/client/client.py", line 782, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1451, in serialize
storage = self.storage.build() # type: Optional[Storage]
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 360, in build
self._build_image(push=push)
File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 427, in _build_image
raise ValueError(
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
I thought that this bit was the most interesting as I have ensured that my backend is server:File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 453, in _request
raise ClientError(
prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
Any ideas as to what has went wrong?josh
01/28/2021, 3:02 PMMatthew Blau
01/28/2021, 3:06 PMwith Flow("example",
storage = Docker(dockerfile="/home/mblau/projects/integration/Dockerfile", ignore_healthchecks= True
)) as flow:
main()
flow.run_config = DockerRun(image = "integration_mosaic_integration:latest")
flow.register(project_name="test")
josh
01/28/2021, 3:07 PMMatthew Blau
01/28/2021, 3:08 PM