h

    Hui Zheng

    1 year ago
    Hello, @Kyle Moon-Wright We ran into a strange issue with Flow to prefect-cloud deployment.The context 1. We use
    flow.storage = Docker()
    for build our flow deployment. 2. we then use
    flow.storage.build(push=False)
    to build the docker container locally and test it run locally. 3. lastly, we use
    flow.register()
    to deploy the flow to prefect-cloud All 3 steps works fine on my local machine. I could deploy the flow to prefect-cloud successfully. My colleagues wants to do the same on their local machine, however, when two my colleagues did it, they ran into a healthcheck issue at
    step 3
    (See issue detail in the thread). we are using the same code base and using the same build-context environment. We have setup another docker container in which we build and deploy prefect flow, so that it ensures that we all have the same run-time and libraries when doing build-and-deploy. This is a high priority issue for us, because currently only I could do the flow deployment. It put high risk on our production downtime if my colleagues could not path and deploy the prefect flow when there is emergency incident situation. cc: @jars @Julie Sturgeon prefect version:
    prefecthq/prefect:0.13.15-python3.8
    error log
    Step 25/25 : RUN python /opt/prefect/healthcheck.py '["/app/flow.py"]' '(3, 8)'
     ---> Running in f50de3703330
    Beginning health checks...
    System Version check: OK
    Traceback (most recent call last):
      File "/opt/prefect/healthcheck.py", line 145, in <module>
        flows = import_flow_from_script_check(flow_file_paths)
      File "/opt/prefect/healthcheck.py", line 59, in import_flow_from_script_check
        flows.append(extract_flow_from_file(file_path=flow_file_path))
      File "/usr/local/lib/python3.8/site-packages/prefect/utilities/storage.py", line 76, in extract_flow_from_file
        exec(contents, exec_vals)
      File "<string>", line 16, in <module>
      File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore.py", line 18, in <module>
        from google.cloud.firestore_v1 import __version__
      File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/__init__.py", line 21, in <module>
        from google.cloud.firestore_v1 import types
      File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/types.py", line 29, in <module>
        from google.cloud.firestore_v1.proto import common_pb2
      File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/proto/common_pb2.py", line 24, in <module>
        create_key=_descriptor._internal_create_key,
    AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
    Removing intermediate container f50de3703330
    The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/app/flow.py"]' '(3, 8)'' returned a non-zero code: 1
    Traceback (most recent call last):
      File "build_and_register.py", line 154, in <module>
        flow.register(
      File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1644, in register
        registered_flow = client.register(
      File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 771, in register
        serialized_flow = flow.serialize(build=build)  # type: Any
      File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1460, in serialize
        storage = self.storage.build()  # type: Optional[Storage]
      File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 351, in build
        self._build_image(push=push)
      File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 417, in _build_image
        raise ValueError(
    ValueError: Your docker image failed to build!  Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
    nicholas

    nicholas

    1 year ago
    Hi @Hui Zheng - this looks to be a problem with your colleague's version of protobuf. Can you confirm that you have matching versions of that package? They may be using an old version.
    Just as an FYI, the Prefect engineers use a support rotation for a number of reasons but we do our best to make sure every question is addressed as soon as possible; unless it's an issue you've specifically discussed with someone that you want to copy them on, I'd ask that you don't @ specific Prefect engineers (we'll pull in others as needed!).
    h

    Hui Zheng

    1 year ago
    thank you @nicholas. thank you for answering.
    yes, I think my colleagues are using the same version of
    protobuf
    because we do the flow building inside the same docker container.
    nicholas

    nicholas

    1 year ago
    Hm. This SO thread points to that error being version-specific so it might be worth execing into both containers to check the version. It's possible that's a package that doesn't have the correct version pinned, which could lead to differences in pip installs.
    h

    Hui Zheng

    1 year ago
    thank you, Nicholas, I will ask my colleagues to try that. cc: @Julie Sturgeon @jars
    just saying that we have
    pip install --upgrade pip
    in our build-flow-docker-container, so every time when we build the flow, the version shall be up-to-date
    nicholas

    nicholas

    1 year ago
    Understood @Hui Zheng - let me know what you find,;this is a bit of a tough one to debug from our side
    h

    Hui Zheng

    1 year ago
    Hello, We found something interesting, indeed. the library versions are different between my colleagues and mine flow docker images, though we build them the same way. hers
    root@e57d800e4155:/app# pip show google-cloud-bigquery
    Name: google-cloud-bigquery
    Version: 1.25.0
    Summary: Google BigQuery API client library
    Home-page: <https://github.com/googleapis/python-bigquery>
    Author: Google LLC
    Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
    License: Apache 2.0
    Location: /usr/local/lib/python3.8/site-packages
    Requires: six, google-auth, google-api-core, google-resumable-media, google-cloud-core, protobuf
    Required-by: dbt-bigquery
    root@e57d800e4155:/app# pip show google-cloud-bigquery
    Name: google-cloud-bigquery
    Version: 1.25.0
    Summary: Google BigQuery API client library
    Home-page: <https://github.com/googleapis/python-bigquery>
    Author: Google LLC
    Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
    License: Apache 2.0
    Location: /usr/local/lib/python3.8/site-packages
    Requires: google-api-core, protobuf, google-auth, six, google-cloud-core, google-resumable-media
    Required-by: dbt-bigquery
    [3:27 PM] root@e57d800e4155:/app# pip show dbt-bigquery
    Name: dbt-bigquery
    Version: 0.18.0
    Summary: The bigquery adapter plugin for dbt (data build tool)
    Home-page: <https://github.com/fishtown-analytics/dbt>
    Author: Fishtown Analytics
    Author-email: <mailto:info@fishtownanalytics.com|info@fishtownanalytics.com>
    License: UNKNOWN
    Location: /usr/local/lib/python3.8/site-packages
    Requires: dbt-core, google-cloud-core, googleapis-common-protos, six, protobuf, google-api-core, google-cloud-bigquery
    Required-by:
    mine
    `Hui Z root@6885cce5a370:/app# pip show google-cloud-bigquery
    Name: google-cloud-bigquery
    Version: 2.4.0
    Summary: Google BigQuery API client library
    Home-page: <https://github.com/googleapis/python-bigquery>
    Author: Google LLC
    Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
    License: Apache 2.0
    Location: /usr/local/lib/python3.8/site-packages
    Requires: protobuf, proto-plus, six, google-api-core, google-resumable-media, google-cloud-core
    Required-by: dbt-bigquery
    [3:34 PM]
    
    5:40 PM
    Hui Z root@6885cce5a370:/app#  pip show dbt-bigquery
    Name: dbt-bigquery
    Version: 0.18.0
    Summary: The bigquery adapter plugin for dbt (data build tool)
    Home-page: <https://github.com/fishtown-analytics/dbt>
    Author: Fishtown Analytics
    Author-email: <mailto:info@fishtownanalytics.com|info@fishtownanalytics.com>
    License: UNKNOWN
    Location: /usr/local/lib/python3.8/site-packages
    Requires: google-api-core, dbt-core, googleapis-common-protos, google-cloud-core, protobuf, six, google-cloud-bigquery
    Required-by:
    still investigating what it could be built differently.