Hui Zheng
12/04/2020, 6:38 PMflow.storage = Docker()
for build our flow deployment.
2. we then use flow.storage.build(push=False)
to build the docker container locally and test it run locally.
3. lastly, we use flow.register()
to deploy the flow to prefect-cloud
All 3 steps works fine on my local machine. I could deploy the flow to prefect-cloud successfully. My colleagues wants to do the same on their local machine, however, when two my colleagues did it, they ran into a healthcheck issue at step 3
(See issue detail in the thread).
we are using the same code base and using the same build-context environment. We have setup another docker container in which we build and deploy prefect flow, so that it ensures that we all have the same run-time and libraries when doing build-and-deploy. This is a high priority issue for us, because currently only I could do the flow deployment. It put high risk on our production downtime if my colleagues could not path and deploy the prefect flow when there is emergency incident situation.
cc: @jars @Julie Sturgeon
prefect version:
prefecthq/prefect:0.13.15-python3.8
Step 25/25 : RUN python /opt/prefect/healthcheck.py '["/app/flow.py"]' '(3, 8)'
---> Running in f50de3703330
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
File "/opt/prefect/healthcheck.py", line 145, in <module>
flows = import_flow_from_script_check(flow_file_paths)
File "/opt/prefect/healthcheck.py", line 59, in import_flow_from_script_check
flows.append(extract_flow_from_file(file_path=flow_file_path))
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/storage.py", line 76, in extract_flow_from_file
exec(contents, exec_vals)
File "<string>", line 16, in <module>
File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore.py", line 18, in <module>
from google.cloud.firestore_v1 import __version__
File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/__init__.py", line 21, in <module>
from google.cloud.firestore_v1 import types
File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/types.py", line 29, in <module>
from google.cloud.firestore_v1.proto import common_pb2
File "/usr/local/lib/python3.8/site-packages/google/cloud/firestore_v1/proto/common_pb2.py", line 24, in <module>
create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
Removing intermediate container f50de3703330
The command '/bin/sh -c python /opt/prefect/healthcheck.py '["/app/flow.py"]' '(3, 8)'' returned a non-zero code: 1
Traceback (most recent call last):
File "build_and_register.py", line 154, in <module>
flow.register(
File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1644, in register
registered_flow = client.register(
File "/usr/local/lib/python3.8/site-packages/prefect/client/client.py", line 771, in register
serialized_flow = flow.serialize(build=build) # type: Any
File "/usr/local/lib/python3.8/site-packages/prefect/core/flow.py", line 1460, in serialize
storage = self.storage.build() # type: Optional[Storage]
File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 351, in build
self._build_image(push=push)
File "/usr/local/lib/python3.8/site-packages/prefect/environments/storage/docker.py", line 417, in _build_image
raise ValueError(
ValueError: Your docker image failed to build! Your flow might have failed one of its deployment health checks - please ensure that all necessary files and dependencies have been included.
nicholas
Hui Zheng
12/04/2020, 7:51 PMprotobuf
because we do the flow building inside the same docker container.nicholas
Hui Zheng
12/04/2020, 8:00 PMpip install --upgrade pip
in our build-flow-docker-container, so every time when we build the flow, the version shall be up-to-datenicholas
Hui Zheng
12/08/2020, 12:18 AMroot@e57d800e4155:/app# pip show google-cloud-bigquery
Name: google-cloud-bigquery
Version: 1.25.0
Summary: Google BigQuery API client library
Home-page: <https://github.com/googleapis/python-bigquery>
Author: Google LLC
Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
License: Apache 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: six, google-auth, google-api-core, google-resumable-media, google-cloud-core, protobuf
Required-by: dbt-bigquery
root@e57d800e4155:/app# pip show google-cloud-bigquery
Name: google-cloud-bigquery
Version: 1.25.0
Summary: Google BigQuery API client library
Home-page: <https://github.com/googleapis/python-bigquery>
Author: Google LLC
Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
License: Apache 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: google-api-core, protobuf, google-auth, six, google-cloud-core, google-resumable-media
Required-by: dbt-bigquery
[3:27 PM] root@e57d800e4155:/app# pip show dbt-bigquery
Name: dbt-bigquery
Version: 0.18.0
Summary: The bigquery adapter plugin for dbt (data build tool)
Home-page: <https://github.com/fishtown-analytics/dbt>
Author: Fishtown Analytics
Author-email: <mailto:info@fishtownanalytics.com|info@fishtownanalytics.com>
License: UNKNOWN
Location: /usr/local/lib/python3.8/site-packages
Requires: dbt-core, google-cloud-core, googleapis-common-protos, six, protobuf, google-api-core, google-cloud-bigquery
Required-by:
mine
`Hui ZÂ root@6885cce5a370:/app# pip show google-cloud-bigquery
Name: google-cloud-bigquery
Version: 2.4.0
Summary: Google BigQuery API client library
Home-page: <https://github.com/googleapis/python-bigquery>
Author: Google LLC
Author-email: <mailto:googleapis-packages@google.com|googleapis-packages@google.com>
License: Apache 2.0
Location: /usr/local/lib/python3.8/site-packages
Requires: protobuf, proto-plus, six, google-api-core, google-resumable-media, google-cloud-core
Required-by: dbt-bigquery
[3:34 PM]
5:40 PM
Hui ZÂ root@6885cce5a370:/app# pip show dbt-bigquery
Name: dbt-bigquery
Version: 0.18.0
Summary: The bigquery adapter plugin for dbt (data build tool)
Home-page: <https://github.com/fishtown-analytics/dbt>
Author: Fishtown Analytics
Author-email: <mailto:info@fishtownanalytics.com|info@fishtownanalytics.com>
License: UNKNOWN
Location: /usr/local/lib/python3.8/site-packages
Requires: google-api-core, dbt-core, googleapis-common-protos, google-cloud-core, protobuf, six, google-cloud-bigquery
Required-by: