https://prefect.io logo
Channels
Powered by
m

Matthew Blau

01/27/2021, 9:01 PM
Hello all, I have made some progress with using docker storage in my flow, however, I am now getting this error:
Copy code
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
The full traceback is
Copy code
Step 12/22 : WORKDIR /app
 ---> Running in b820b907ef61
Removing intermediate container b820b907ef61
 ---> fee83981b109
Step 13/22 : RUN groupadd --gid ${GID} ${USERNAME}     && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME}     && chown -R ${USERNAME} /app     && chgrp -R ${USERNAME} /app
 ---> Running in 7f6c8b47c9e3
groupadd: invalid group ID 'rainbow'

Removing intermediate container 7f6c8b47c9e3
The command '/bin/sh -c groupadd --gid ${GID} ${USERNAME}     && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME}     && chown -R ${USERNAME} /app     && chgrp -R ${USERNAME} /app' returned a non-zero code: 3
Traceback (most recent call last):
  File "integration.py", line 11, in <module>
    from change_password import *
  File "/home/mblau/projects/experian-integration/change_password.py", line 5, in <module>
    from integration import modify_config
  File "/home/mblau/projects/experian-integration/integration.py", line 338, in <module>
    flow.register(project_name="test")
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
    registered_flow = client.register(
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/client/client.py", line 782, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1451, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 360, in build
    self._build_image(push=push)
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 427, in _build_image
    raise ValueError(
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
what am I doing wrong?
j

josh

01/27/2021, 9:20 PM
@Matthew Blau this error leads me to think that your flow is dependent on other files or other python packages that you aren’t building into your Docker storage. If there are local files that you are importing from you can add them to your image using the 
files
kwarg and if there are extra pip-installable dependencies they can be specified with 
python_dependencies
m

Matthew Blau

01/27/2021, 9:23 PM
@josh Yes, we have local files that we are importing as our own custom python modules. change_password.py and integrations_lookups.py
@josh I see that files takes a dictionary like files={'src' : 'desc'}, meaning it is wanting a directory to look for, yes? If that is the case, what would the desc be?
j

josh

01/27/2021, 9:29 PM
So the src is the path to the file on your local machine and dest is where you want to place it in the image. The example in the documentation takes the files and puts them into a 
/modules/
directory and then adds that directory to the PYTHONPATH so they can be imported. The dest is up to you but just not you need to add it to the PYTHONPATH so your flow can import them
m

Matthew Blau

01/27/2021, 9:35 PM
@josh so in my case, I should be copying them to /app? In the Dockerfile I have
Copy code
COPY *.py /app/
COPY ./sql/* /app/sql/
WORKDIR /app
@josh this is what I have as my relevant bit of code presently:
Copy code
with Flow("example",
            storage = Docker(dockerfile="/home/mblau/projects/experian-integration/Dockerfile", 
                             files={'/home/mblau/projects/experian-integration/integration.py' : '/app', 
                            '/home/mblau/projects/experian-integration/integration_lookups.py' : '/app', 
                            '/home/mblau/projects/experian-integration/change_password.py' : '/app'})
) as flow:
j

josh

01/27/2021, 9:44 PM
Yeah that looks right, not sure if you need to add the file name or not (e.g. 
/app/change_password.py
. All that you need to do is set the python path env like the docs does to add it to the path so it can be imported:
Copy code
env_vars={
        # append modules directory to PYTHONPATH
        "PYTHONPATH": "$PYTHONPATH:app/"
    },
m

Matthew Blau

01/27/2021, 9:59 PM
@josh I have adjusted the code and am getting this as a traceback:
Copy code
Step 11/23 : RUN groupadd --gid ${GID} ${USERNAME}     && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME}     && chown -R ${USERNAME} /app     && chgrp -R ${USERNAME} /app
 ---> Running in db46d0aafd98
groupadd: invalid group ID 'rainbow'

Removing intermediate container db46d0aafd98
The command '/bin/sh -c groupadd --gid ${GID} ${USERNAME}     && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME}     && chown -R ${USERNAME} /app     && chgrp -R ${USERNAME} /app' returned a non-zero code: 3
Traceback (most recent call last):
  File "integration.py", line 11, in <module>
    from change_password import *
  File "/home/mblau/projects/experian-integration/change_password.py", line 5, in <module>
    from integration import modify_config
  File "/home/mblau/projects/experian-integration/integration.py", line 349, in <module>
    flow.register(project_name="test")
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
    registered_flow = client.register(
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/client/client.py", line 782, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1451, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 360, in build
    self._build_image(push=push)
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 427, in _build_image
    raise ValueError(
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
Code has been adjusted to
Copy code
with Flow("example",
            storage = Docker(dockerfile="/home/mblau/projects/experian-integration/Dockerfile", 
                             files={'/home/mblau/projects/experian-integration/integration.py' : 
                                 'experian-integration/integration.py/app', 
                            '/home/mblau/projects/experian-integration/integration_lookups.py' : 
                                'integration_lookups.py/app', 
                            '/home/mblau/projects/experian-integration/change_password.py' : 
                                'change_password.py/app'},
                             env_vars={
        # append modules directory to PYTHONPATH
        "PYTHONPATH": "$PYTHONPATH:app/"
    },)
j

josh

01/27/2021, 10:05 PM
Why are you writing the dest as 
file.py/app
and not 
app/file.py
?
👍 1
m

Matthew Blau

01/27/2021, 10:06 PM
Because I've been looking at this way too long and am making errors lol
🙂 1
Thanks for the catch! I'll try that
@josh changed the directory to be app/file.py and the error remains
j

josh

01/27/2021, 10:20 PM
Oh hmm looking at it now I think the cause may be the use of a Dockerfile. The 
files
attribute is used when not using a custom Dockerfile. Working with a dockerfile makes the assumption that your dockerfile copies over the appropriate files. In your Dockerfile maybe adding this will help:
Copy code
COPY file.py /app/file.py
..for all your files..

ENV PYTHONPATH="${PYTHONPATH}:/app"
Instead of using env and files on the Docker storage object
m

Matthew Blau

01/27/2021, 10:29 PM
@josh closer!
Copy code
Step 13/24 : ENV PYTHONPATH = "${PYTHONPATH"}:/app"
failed to process "= \"${PYTHONPATH\"}:/app\"": missing ':' in substitution
is the new error
think I fixed that now the only error remaining is:
Copy code
Removing intermediate container 9d8046a5b267
 ---> 6c27481596d3
Step 15/24 : RUN groupadd --gid ${GID} ${USERNAME}     && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME}     && chown -R ${USERNAME} /app     && chgrp -R ${USERNAME} /app
 ---> Running in d08baee33e77
groupadd: invalid group ID 'rainbow'

Removing intermediate container d08baee33e77
The command '/bin/sh -c groupadd --gid ${GID} ${USERNAME}     && useradd --uid ${UID} --gid ${GID} --shell /bin/bash -M ${USERNAME}     && chown -R ${USERNAME} /app     && chgrp -R ${USERNAME} /app' returned a non-zero code: 3
@josh I have determined that I need to pass build args to the file and that the build args need to be a dict. The build args to build the dockerfile outside of prefect are
Copy code
docker-compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
and I am unsure of what Prefect is expecting for the dict. I know it's after 5pm as well on the east coast of the US so I hope I am not bothering you too much. I can always wait til tomorrow for an answer, but wanted to put my latest finds out there at least 🙂
j

josh

01/27/2021, 10:54 PM
Docker storage supports build args like this:
Copy code
Docker(build_kwargs={"UID": ..., "GID": ...})
m

Matthew Blau

01/28/2021, 1:29 PM
Hello @josh I had removed the bits of the Dockerfile that required build args as I could not get the build args to work properly, resolved some circular import issues, and have come across this error in the traceback:
Copy code
Step 22/22 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/example.prefect"]' '(3, 8)'
 ---> Running in d892350e6f62
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 450, in _request
    json_resp = response.json()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 147, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 40, in cloudpickle_deserialization_check
    flows.append(cloudpickle.load(f))
  File "/app/integration.py", line 341, in <module>
    flow.register(project_name="test")
  File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
    registered_flow = client.register(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 773, in register
    project = self.graphql(query_project).data.project  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 297, in graphql
    result = <http://self.post|self.post>(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 212, in post
    response = self._request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 453, in _request
    raise ClientError(
prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.

Removing intermediate container d892350e6f62
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/opt/prefect/flows/example.prefect"]' '(3, 8)'' returned a non-zero code: 1
Traceback (most recent call last):
  File "integration.py", line 11, in <module>
    from change_password import *
  File "/home/mblau/projects/experian-integration/change_password.py", line 5, in <module>
    from integration import *
  File "/home/mblau/projects/experian-integration/integration.py", line 341, in <module>
    flow.register(project_name="test")
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1665, in register
    registered_flow = client.register(
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/client/client.py", line 782, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/core/flow.py", line 1451, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 360, in build
    self._build_image(push=push)
  File "/home/mblau/.local/lib/python3.8/site-packages/prefect/storage/docker.py", line 427, in _build_image
    raise ValueError(
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
I thought that this bit was the most interesting as I have ensured that my backend is server:
Copy code
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 453, in _request
    raise ClientError(
prefect.utilities.exceptions.ClientError: Malformed response received from Cloud - please ensure that you have an API token properly configured.
Any ideas as to what has went wrong?
j

josh

01/28/2021, 3:02 PM
Do one of the files that your flow depends on happen to have a flow.register in it?
m

Matthew Blau

01/28/2021, 3:06 PM
Hi @josh I only have a pretty simple flow going on right now with 1 file as the flow with the bulk of the integration logic. This is my flow for reference:
Copy code
with Flow("example",
            storage = Docker(dockerfile="/home/mblau/projects/integration/Dockerfile", ignore_healthchecks= True
)) as flow:
    main()
flow.run_config = DockerRun(image = "integration_mosaic_integration:latest")
flow.register(project_name="test")
I had turned on the ignore health checks feature as debug attempt
j

josh

01/28/2021, 3:07 PM
Would you mind opening this as an issue on the repo? It’s easier to debug things on the github interface and this thread is getting long 🙂
m

Matthew Blau

01/28/2021, 3:08 PM
@josh sure!
Powered by