Channels
#prefect-community
Title
a
Adam
02/01/2021, 12:15 PMHello friends, our CI process is suddenly failing due to *errors in importing GCP / BigQuery / Google Core dependencie*s. We haven’t changed anything on our side so curious to know if anyone else is seeing this issue suddenly. Details are in the thread. Any help would be much appreciated!
The errors we are seeing:
Copy code
_________________________________________________________________ ERROR collecting tests/test_credit_reporting.py _________________________________________________________________
ImportError while importing test module '/Users/adam/code/sable-batch/tests/test_credit_reporting.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:13: in <module>
from prefect.tasks.gcp.storage import GCSDownload, GCSUpload, GCSCopy
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/storage.py:6: in <module>
from google.cloud.exceptions import NotFound
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_core-1.5.0-py3.8.egg/google/cloud/exceptions.py:24: in <module>
from google.api_core import exceptions
E ModuleNotFoundError: No module named 'google.api_core'
The above exception was the direct cause of the following exception:
../../.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_credit_reporting.py:6: in <module>
from sable_batch.flows.credit_reporting_payment_status import (
sable_batch/flows/credit_reporting_payment_status.py:9: in <module>
from prefect.tasks.gcp.bigquery import BigQueryTask
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:22: in <module>
raise ImportError(
E ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.
___________________________________________________________________ ERROR collecting tests/test_demographics.py ___________________________________________________________________
ImportError while importing test module '/Users/adam/code/sable-batch/tests/test_demographics.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:13: in <module>
from prefect.tasks.gcp.storage import GCSDownload, GCSUpload, GCSCopy
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/storage.py:6: in <module>
from google.cloud.exceptions import NotFound
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_core-1.5.0-py3.8.egg/google/cloud/exceptions.py:24: in <module>
from google.api_core import exceptions
E ModuleNotFoundError: No module named 'google.api_core'
The above exception was the direct cause of the following exception:
../../.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_demographics.py:1: in <module>
from sable_batch.flows.demographics import create_customer_object
sable_batch/flows/demographics.py:4: in <module>
from prefect.tasks.gcp.bigquery import BigQueryTask
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:22: in <module>
raise ImportError(
E ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.We’re also seeing errors when using
from google.cloud import bigquery Copy code
___________________________________________________________ ERROR collecting tests/test_import_synapse_transactions.py ____________________________________________________________
ImportError while importing test module '/Users/adam/code/sable-batch/tests/test_import_synapse_transactions.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_import_synapse_transactions.py:1: in <module>
from sable_batch.flows.import_synapse_transactions import _calculate_pages_bins
sable_batch/flows/import_synapse_transactions.py:9: in <module>
from google.cloud import bigquery
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_bigquery-1.28.0-py3.8.egg/google/cloud/bigquery/__init__.py:35: in <module>
from google.cloud.bigquery.client import Client
../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_bigquery-1.28.0-py3.8.egg/google/cloud/bigquery/client.py:47: in <module>
import google.api_core.client_options
E ModuleNotFoundError: No module named 'google.api_core'Our `setup.py`:
Copy code
requirements = (
[
"prefect[google,kubernetes,postgres,viz,gcp]==0.13.18",
"pandas",
"requests",
"analytics-python",
"beautifulsoup4",
"synapsepy",
"cloudpickle",
"python-dotenv",
"sqlalchemy",
"google-cloud-bigquery-storage<2.0.0",
"pyarrow",
"phonenumbers",
"pycountry",
"country-converter",
"flatten-dict",
"us",
"pysftp",
"pytest",
"googleads",
"python-gnupg",
"Sift",
"xlsxwriter",
"chardet<4.0.0",
],
)FWIW, I noticed that
google-cloud-bigqueryUpdated to the latest version of prefect did not help unfortuantely
z
Zanie
02/01/2021, 6:08 PM
Hi! We’re looking into this. Could you try adding
google-api-core