https://prefect.io logo
Join Slack
Powered by
Hello, <@UUDFU628J> We ran into a strange issue w...
# prefect-community
h
Hello, @Kyle Moon-Wright We ran into a strange issue with Flow to prefect-cloud deployment. The context 1. We use 
flow.storage = Docker()
for build our flow deployment. 2. we then use 
flow.storage.build(push=False)
to build the docker container locally and test it run locally. 3. lastly, we use 
flow.register()
to deploy the flow to prefect-cloud All 3 steps works fine on my local machine. I could deploy the flow to prefect-cloud successfully. My colleagues wants to do the same on their local machine, however, when two my colleagues did it, they ran into a healthcheck issue at 
step 3
(See issue detail in the thread). we are using the same code base and using the same build-context environment. We have setup another docker container in which we build and deploy prefect flow, so that it ensures that we all have the same run-time and libraries when doing build-and-deploy. This is a high priority issue for us, because currently only I could do the flow deployment. It put high risk on our production downtime if my colleagues could not path and deploy the prefect flow when there is emergency incident situation. cc: @jars @Julie Sturgeon prefect version:
Copy code
prefecthq/prefect:0.13.15-python3.8
error log
Copy code
Step 25/25 : RUN python /opt/prefect/healthcheck.py '["/app/flow.py"]' '(3, 8)'
 ---> Running in f50de3703330
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 145, in <module>
    flows = import_flow_from_script_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 59, in import_flow_from_script_check
    flows.append(extract_flow_from_file(file_path=flow_file_path))
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/storage.py", line 76, in extract_flow_from_file
    exec(contents, exec_vals)
  File "<string>", line 16, in <module>
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore.py", line 18, in <module>
    from google.cloud.firestore_v1 import __version__
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/__init__.py", line 21, in <module>
    from google.cloud.firestore_v1 import types
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/types.py", line 29, in <module>
    from google.cloud.firestore_v1.proto import common_pb2
  File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/proto/common_pb2.py", line 24, in <module>
    create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
Removing intermediate container f50de3703330
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/app/flow.py"]' '(3, 8)'' returned a non-zero code: 1
Traceback (most recent call last):
  File "build_and_register.py", line 154, in <module>
    flow.register(
  File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1644, in register
    registered_flow = client.register(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 771, in register
    serialized_flow = flow.serialize(build=build)  # type: Any
  File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1460, in serialize
    storage = self.storage.build()  # type: Optional[Storage]
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 351, in build
    self._build_image(push=push)
  File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 417, in _build_image
    raise ValueError(
ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
n
Hi @Hui Zheng - this looks to be a problem with your colleague's version of protobuf. Can you confirm that you have matching versions of that package? They may be using an old version.
Just as an FYI, the Prefect engineers use a support rotation for a number of reasons but we do our best to make sure every question is addressed as soon as possible; unless it's an issue you've specifically discussed with someone that you want to copy them on, I'd ask that you don't @ specific Prefect engineers (we'll pull in others as needed!).
h
thank you @nicholas. thank you for answering.
👍 1
yes, I think my colleagues are using the same version of 
protobuf
because we do the flow building inside the same docker container.
n
Hm. This SO thread points to that error being version-specific so it might be worth `exec`ing into both containers to check the version. It's possible that's a package that doesn't have the correct version pinned, which could lead to differences in pip installs.
h
thank you, Nicholas, I will ask my colleagues to try that. cc: @Julie Sturgeon @jars
just saying that we have
Copy code
pip install --upgrade pip
in our build-flow-docker-container, so every time when we build the flow, the version shall be up-to-date
n
Understood @Hui Zheng - let me know what you find,;this is a bit of a tough one to debug from our side
h
Hello, We found something interesting, indeed. the library versions are different between my colleagues and mine flow docker images, though we build them the same way. hers
Copy code
root@e57d800e4155:/app# pip show google-cloud-bigquery
Name: google-cloud-bigquery
Version: 1.25.0
Summary: Google BigQuery API client library
Home-page: <https://github.com/googleapis/python-bigquery>
Author: Google LLC
Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
License: Apache 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: six, google-auth, google-api-core, google-resumable-media, google-cloud-core, protobuf
Required-by: dbt-bigquery
root@e57d800e4155:/app# pip show google-cloud-bigquery
Name: google-cloud-bigquery
Version: 1.25.0
Summary: Google BigQuery API client library
Home-page: <https://github.com/googleapis/python-bigquery>
Author: Google LLC
Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
License: Apache 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: google-api-core, protobuf, google-auth, six, google-cloud-core, google-resumable-media
Required-by: dbt-bigquery
[3:27 PM] root@e57d800e4155:/app# pip show dbt-bigquery
Name: dbt-bigquery
Version: 0.18.0
Summary: The bigquery adapter plugin for dbt (data build tool)
Home-page: <https://github.com/fishtown-analytics/dbt>
Author: Fishtown Analytics
Author-email: <mailto:info@fishtownanalytics.com|info@fishtownanalytics.com>
License: UNKNOWN
Location: /usr/local/lib/python3.8/site-packages
Requires: dbt-core, google-cloud-core, googleapis-common-protos, six, protobuf, google-api-core, google-cloud-bigquery
Required-by:
mine
Copy code
`Hui Z root@6885cce5a370:/app# pip show google-cloud-bigquery
Name: google-cloud-bigquery
Version: 2.4.0
Summary: Google BigQuery API client library
Home-page: <https://github.com/googleapis/python-bigquery>
Author: Google LLC
Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
License: Apache 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: protobuf, proto-plus, six, google-api-core, google-resumable-media, google-cloud-core
Required-by: dbt-bigquery
[3:34 PM]

5:40 PM
Hui Z root@6885cce5a370:/app#  pip show dbt-bigquery
Name: dbt-bigquery
Version: 0.18.0
Summary: The bigquery adapter plugin for dbt (data build tool)
Home-page: <https://github.com/fishtown-analytics/dbt>
Author: Fishtown Analytics
Author-email: <mailto:info@fishtownanalytics.com|info@fishtownanalytics.com>
License: UNKNOWN
Location: /usr/local/lib/python3.8/site-packages
Requires: google-api-core, dbt-core, googleapis-common-protos, google-cloud-core, protobuf, six, google-cloud-bigquery
Required-by:
still investigating what it could be built differently.
Open in Slack
PreviousNext
Powered by