Jean-Baptiste Six
01/25/2022, 2:25 PMfrom dotenv import load_dotenv
load_dotenv()
@task()
def add(x, y=1):
return x + y
with Flow("Test GCSResult", result=GCSResult(bucket='thread-prefect-flows')) as GCSResult_flow:
first_result = add(1, y=2)
second_result = add(x=first_result, y=100)
Without success, however :
from dotenv import load_dotenv
load_dotenv()
def implicit():
from google.cloud import storage
# If you don't specify credentials when constructing the client, the
# client library will look for credentials in the environment.
storage_client = storage.Client()
# Make an authenticated API request
buckets = list(storage_client.list_buckets())
print(buckets)
implicit()
works well, then I'm not sure to understand : "_Make sure your Prefect installation can authenticate to Google's Cloud API_" (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult)
Then, I tried to use Local Secret (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult)
[context.secrets]
GOOGLE_APPLICATION_CREDENTIALS="auth/token.json"
GCP_CREDENTIALS="auth/token.json"
(don't know what to use between GCP_CREDENTIALS (prefect.utiliities.gcp) or GOOGLE_APPLICATION_CREDENTIALS (in the documentation of GCP), anyway I tried both and it didn't work neither.
What did I do wrong?
Please help 🙏Anna Geller
Anna Geller
gsutil ls -r <gs://thread-prefect-flows>
Jean-Baptiste Six
01/25/2022, 2:34 PMgsutil ls -r <gs://thread-prefect-flows>
works wellAnna Geller
prefect agent xxx start --env PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS="your_creds"
Jean-Baptiste Six
01/25/2022, 2:36 PMJean-Baptiste Six
01/25/2022, 2:39 PMdef test_GCSResult():
GCSResult_flow.run()
Anna Geller
Jean-Baptiste Six
01/25/2022, 2:48 PMAnna Geller
Jean-Baptiste Six
01/25/2022, 2:53 PMJean-Baptiste Six
01/25/2022, 4:42 PMUnexpected error: AttributeError("'str' object has no attribute 'keys'")
Right now I have the following secret :
GCP_CREDENTIALS = "auth/token.json"
where is stored my credentials for GCP. But when I see the error, it make me think that I need to directly give the content of the json file in the secret, is that correct ?Kevin Kho
GOOGLE_APPLICATION_CREDENTIALS
, you point it to the json. If you use the GCP_CREDENTIALS
, this is a secret that contains the content of the JSONKevin Kho
GOOGLE_APPLICATION_CREDENTIALS
is google convention. GCP_CREDENTIALS
is a Prefect convention because we would store the content of the JSON in CloudKevin Kho
GOOGLE_APPLICATION_CREDENTIALS
, you just need the .json
file in the execution environmentJean-Baptiste Six
01/26/2022, 8:49 AMJean-Baptiste Six
01/26/2022, 9:01 AMJean-Baptiste Six
01/26/2022, 9:21 AMAnna Geller
Jean-Baptiste Six
01/26/2022, 2:14 PMKevin Kho
GOOGLE_APPLICATION_CREDENTIALS
should not be a Secret stored with us. It should be an environment variable stored in your execution environment