https://prefect.io logo
j

Jonas Hanfland

10/06/2020, 4:43 PM
Hey guys, I am getting an exception when trying to run a
BigQueryTask
. This is the task I am trying to run:
Copy code
BigQueryTask(
    name="Get all statement IDs",
    query="""
        SELECT * FROM temporary_tables.statements
    """,
    max_retries=3,
    retry_delay=timedelta(seconds=30),
    to_dataframe=True,
)
and it throws this exception: (see thread) All the times this task has been run in the past it passed just fine. Does anybody have an idea what might be the issue? Thanks very much in advance!
c

Chris White

10/06/2020, 4:46 PM
Hi Jonas - would you mind moving your traceback into this thread? It’s very large and takes up the whole slack window. Otherwise, this is most likely caused by having an unsupported version of google’s bigquery client in your environment. What version do you have installed?
j

Jonas Hanfland

10/06/2020, 4:48 PM
Copy code
[2020-10-06 15:39:25] ERROR - prefect.TaskRunner | Unexpected error: AttributeError("'BigQueryReadClient' object has no attribute 'transport'")
Traceback (most recent call last):
  File "/home/venv/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 1545, in to_arrow
    for record_batch in self._to_arrow_iterable(
  File "/home/venv/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 1435, in _to_page_iterable
    for item in bqstorage_download():
  File "/home/venv/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 629, in _download_table_bqstorage
    from google.cloud import bigquery_storage_v1beta1
ImportError: cannot import name 'bigquery_storage_v1beta1' from 'google.cloud' (unknown location)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/venv/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/home/venv/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 824, in get_task_run_state
    value = timeout_handler(
  File "/home/venv/lib/python3.8/site-packages/prefect/utilities/executors.py", line 188, in timeout_handler
    return fn(*args, **kwargs)
  File "/home/venv/lib/python3.8/site-packages/prefect/utilities/tasks.py", line 449, in method
    return run_method(self, *args, **kwargs)
  File "/home/venv/lib/python3.8/site-packages/prefect/tasks/gcp/bigquery.py", line 181, in run
    return query_job.result().to_dataframe()
  File "/home/venv/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 1725, in to_dataframe
    record_batch = self.to_arrow(
  File "/home/venv/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 1562, in to_arrow
    bqstorage_client.transport.channel.close()
AttributeError: 'BigQueryReadClient' object has no attribute 'transport'
👍 1
Copy code
Name: google-cloud-bigquery-storage
Version: 2.0.0
c

Chris White

10/06/2020, 4:52 PM
ah ha! At this moment version 2.0.0 is not supported: https://github.com/PrefectHQ/prefect/blob/master/setup.py#L44-L45 It seems we’ll need to do some digging to figure out how to support 2.0+
🙏 1
j

Jonas Hanfland

10/06/2020, 4:56 PM
Amazing. I will try to use an older version for now. Thank you so much for the crazy fast solution!!
c

Chris White

10/06/2020, 4:57 PM
Anytime! Glad to help 👍