https://prefect.io logo
Title
s

Steven Trimboli

12/07/2022, 6:09 PM
Hi - I have had trouble passing credentials to validate bigquery for some time now. I pasted the json into a GCP Credentials block Then I try validating in this way:
from prefect_gcp import GcpCredentials

gcp_credentials_block = GcpCredentials.load("bq-credentials")
bigquery_client = bigquery.Client(credentials=gcp_credentials_block)
I receive this error message: "ValueError: This library only supports credentials from google-auth-library-python. See https://google-auth.readthedocs.io/en/latest/ for help on authentication with this library." How can I pass credentials to GCP successfully?
1
Block screenshot if it helps
Do I need to use the prefect_gcp.bigquery library to leverage this block?
k

Kevin Grismore

12/07/2022, 6:21 PM
I believe the bigquery client is expecting a
Credentials
object. The data stored in the GcpCredentials block is
service_account_info
. If you install the
google-auth
python package, you can do something like:
from google.oauth2.service_account import Credentials
from prefect_gcp import GcpCredentials

gcp_credentials_block = GcpCredentials.load("bq-credentials")
creds = Credentials.from_service_account_info(gcp_credentials_block.service_account_info)
bigquery_client = bigquery.Client(credentials=creds)
Alternatively, you can pass the
gcp_credentials_block
directly into any of the
prefect_gcp.bigquery
tasks without needing to make your own bigquery client, but you'll be limited to the functionality provided by the task collection
a

Andrew Huang

12/07/2022, 6:45 PM
There’s a difference between GcpCredentials and Google’s auth credentials. However, GcpCredentials provides a method, get_credentials_from_service_account to get Google’s auth credentials. So you can do:
from prefect_gcp import GcpCredentials

gcp_credentials_block = GcpCredentials.load("bq-credentials")
google_auth_credentials = gcp_credentials_block.get_credentials_from_service_account()
bigquery_client = bigquery.Client(credentials=google_auth_credentials)
Alternatively, you can simply get the bigquery client directly from gcp_credentials
gcp_credentials_block = GcpCredentials.load("bq-credentials").get_bigquery_client()
or best practice is simply to use one of the built-in tasks: https://github.com/PrefectHQ/prefect-gcp/blob/main/prefect_gcp/bigquery.py
k

Kevin Grismore

12/07/2022, 6:46 PM
^^^ 🙌
didn't realize
.get_credentials_from_service_account()
existed. This'll help me too. Thanks!
🙌 2
s

Steven Trimboli

12/07/2022, 6:48 PM
I receive this error "'coroutine' object has no attribute 'get_credentials_from_service_account'" am i missing a download?
and reason i am taking this route is because outside of bigquery - also need to pass GDrive credentials
a

Andrew Huang

12/07/2022, 6:56 PM
are you using a notebook; if so, you need to await
gcp_credentials_block = await GcpCredentials.load("bq-credentials")
or, you can simply put everything into a flow, and it should work without the await
s

Steven Trimboli

12/07/2022, 6:57 PM
i am using a notebook for building...that is great caveat to know. thank you, that worked now!
🙌 2
appreciate the help @Andrew Huang and @Kevin Grismore!
a

Andrew Huang

12/07/2022, 6:58 PM
yep it’s an issue here I’ll see if I can fix it soon because I also work in notebooks and it’s a bother
👍 2