Adam

    Adam

    1 year ago
    Hello friends, our CI process is suddenly failing due to errors in importing GCP / BigQuery / Google Core dependencies. We haven’t changed anything on our side so curious to know if anyone else is seeing this issue suddenly. Details are in the thread. Any help would be much appreciated!
    The errors we are seeing:
    _________________________________________________________________ ERROR collecting tests/test_credit_reporting.py _________________________________________________________________
    ImportError while importing test module '/Users/adam/code/sable-batch/tests/test_credit_reporting.py'.
    Hint: make sure your test modules/packages have valid Python names.
    Traceback:
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:13: in <module>
        from prefect.tasks.gcp.storage import GCSDownload, GCSUpload, GCSCopy
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/storage.py:6: in <module>
        from google.cloud.exceptions import NotFound
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_core-1.5.0-py3.8.egg/google/cloud/exceptions.py:24: in <module>
        from google.api_core import exceptions
    E   ModuleNotFoundError: No module named 'google.api_core'
    
    The above exception was the direct cause of the following exception:
    ../../.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py:127: in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    tests/test_credit_reporting.py:6: in <module>
        from sable_batch.flows.credit_reporting_payment_status import (
    sable_batch/flows/credit_reporting_payment_status.py:9: in <module>
        from prefect.tasks.gcp.bigquery import BigQueryTask
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:22: in <module>
        raise ImportError(
    E   ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.
    ___________________________________________________________________ ERROR collecting tests/test_demographics.py ___________________________________________________________________
    ImportError while importing test module '/Users/adam/code/sable-batch/tests/test_demographics.py'.
    Hint: make sure your test modules/packages have valid Python names.
    Traceback:
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:13: in <module>
        from prefect.tasks.gcp.storage import GCSDownload, GCSUpload, GCSCopy
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/storage.py:6: in <module>
        from google.cloud.exceptions import NotFound
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_core-1.5.0-py3.8.egg/google/cloud/exceptions.py:24: in <module>
        from google.api_core import exceptions
    E   ModuleNotFoundError: No module named 'google.api_core'
    
    The above exception was the direct cause of the following exception:
    ../../.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py:127: in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    tests/test_demographics.py:1: in <module>
        from sable_batch.flows.demographics import create_customer_object
    sable_batch/flows/demographics.py:4: in <module>
        from prefect.tasks.gcp.bigquery import BigQueryTask
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/prefect-0.13.18-py3.8.egg/prefect/tasks/gcp/__init__.py:22: in <module>
        raise ImportError(
    E   ImportError: Using `prefect.tasks.gcp` requires Prefect to be installed with the "gcp" extra.
    We’re also seeing errors when using
    from google.cloud import bigquery
    directly so perhaps the issue is downstream on google’s libraries?
    ___________________________________________________________ ERROR collecting tests/test_import_synapse_transactions.py ____________________________________________________________
    ImportError while importing test module '/Users/adam/code/sable-batch/tests/test_import_synapse_transactions.py'.
    Hint: make sure your test modules/packages have valid Python names.
    Traceback:
    ../../.pyenv/versions/3.8.5/lib/python3.8/importlib/__init__.py:127: in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    tests/test_import_synapse_transactions.py:1: in <module>
        from sable_batch.flows.import_synapse_transactions import _calculate_pages_bins
    sable_batch/flows/import_synapse_transactions.py:9: in <module>
        from google.cloud import bigquery
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_bigquery-1.28.0-py3.8.egg/google/cloud/bigquery/__init__.py:35: in <module>
        from google.cloud.bigquery.client import Client
    ../../.virtualenvs/sable-batch/lib/python3.8/site-packages/google_cloud_bigquery-1.28.0-py3.8.egg/google/cloud/bigquery/client.py:47: in <module>
        import google.api_core.client_options
    E   ModuleNotFoundError: No module named 'google.api_core'
    Our setup.py:
    requirements = (
        [
            "prefect[google,kubernetes,postgres,viz,gcp]==0.13.18",
            "pandas",
            "requests",
            "analytics-python",
            "beautifulsoup4",
            "synapsepy",
            "cloudpickle",
            "python-dotenv",
            "sqlalchemy",
            "google-cloud-bigquery-storage<2.0.0",
            "pyarrow",
            "phonenumbers",
            "pycountry",
            "country-converter",
            "flatten-dict",
            "us",
            "pysftp",
            "pytest",
            "googleads",
            "python-gnupg",
            "Sift",
            "xlsxwriter",
            "chardet<4.0.0",
        ],
    )
    FWIW, I noticed that
    google-cloud-bigquery
    have released a new version a few days ago, perhaps related?
    Updated to the latest version of prefect did not help unfortuantely
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! We’re looking into this. Could you try adding
    google-api-core
    to your requirements list?