Hi :slightly_smiling_face: I have an issue to use ...
# ask-community
j
Hi 🙂 I have an issue to use _*GCSResult, t*_he results are not stored (but there isn't any error raised). First, I tried to use environment variable :
Copy code
from dotenv import load_dotenv
load_dotenv()

@task()
def add(x, y=1):
    return x + y

with Flow("Test GCSResult", result=GCSResult(bucket='thread-prefect-flows')) as GCSResult_flow:
    first_result = add(1, y=2)
    second_result = add(x=first_result, y=100)
Without success, however :
Copy code
from dotenv import load_dotenv
load_dotenv()

def implicit():
    from google.cloud import storage

    # If you don't specify credentials when constructing the client, the
    # client library will look for credentials in the environment.
    storage_client = storage.Client()

    # Make an authenticated API request
    buckets = list(storage_client.list_buckets())
    print(buckets)

implicit()
works well, then I'm not sure to understand : "_Make sure your Prefect installation can authenticate to Google's Cloud API_" (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult) Then, I tried to use Local Secret (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult)
Copy code
[context.secrets]
GOOGLE_APPLICATION_CREDENTIALS="auth/token.json"
GCP_CREDENTIALS="auth/token.json"
(don't know what to use between GCP_CREDENTIALS (prefect.utiliities.gcp) or GOOGLE_APPLICATION_CREDENTIALS (in the documentation of GCP), anyway I tried both and it didn't work neither. What did I do wrong? Please help 🙏
a
1. Are you on Prefect Cloud or Server? 2. What agent do you use? GCP credentials must be available on your agent
if you SSH to your agent’s machine and type this in your terminal, does it work? This is how you can test whether your agent has the right permissions:
Copy code
gsutil ls -r <gs://thread-prefect-flows>
j
Copy code
gsutil ls -r <gs://thread-prefect-flows>
works well
a
you can also set it when you start your agent:
Copy code
prefect agent xxx start --env PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS="your_creds"
j
For the first question, I use Prefect Cloud, but I want to make it work in local For the second one, i don't know, what are the different kind of agents ?
I'm using a local agent. To run the flow, I just do :
Copy code
def test_GCSResult():
    GCSResult_flow.run()
a
to use results, you need to deploy your flow to a backend, i.e. you would need to register your flow and trigger it e.g. from the UI/CLI/API.
j
I see ! It is not possible to test GCSResult in local then ?
a
It depends on what do you consider local 🙂 you can use it with your local agent and then effectively everything runs locally, but the execution (incl. result handling) is governed by the Cloud API
🙏 1
j
@Luis Arias FYI
@Anna Geller I have investigated, and I have the following error :
Copy code
Unexpected error: AttributeError("'str' object has no attribute 'keys'")
Right now I have the following secret :
Copy code
GCP_CREDENTIALS = "auth/token.json"
where is stored my credentials for GCP. But when I see the error, it make me think that I need to directly give the content of the json file in the secret, is that correct ?
k
So if you use
GOOGLE_APPLICATION_CREDENTIALS
, you point it to the json. If you use the
GCP_CREDENTIALS
, this is a secret that contains the content of the JSON
GOOGLE_APPLICATION_CREDENTIALS
is google convention.
GCP_CREDENTIALS
is a Prefect convention because we would store the content of the JSON in Cloud
🙏 1
If you use
GOOGLE_APPLICATION_CREDENTIALS
, you just need the
.json
file in the execution environment
j
Thanks @Kevin Kho!
How could I use the secrets defined in the UI ?
Btw, I succeeded to make it work with GCP_CREDENTIALS (by converting the json into TOML file into ~/.prefect/config.toml (not super intuitive)) But the secret GOOGLE_APPLICATION_CREDENTIALS = "auth/token.json" doesn't seem to work, is it a bug ?
a
As Kevin explained, GCP_CREDENTIALS is the Prefect-specific implementation so that you can store the secret in the Cloud Secret backend. If that works for you, no need to worry about GOOGLE_APPLICATION_CREDENTIALS which are intended to only reference the path to the JSON file describing permissions of the service account.
j
I know 👍 but if GCP_CREDENTIALS is not defined, GOOGLE_APPLICATION_CREDENTIALS should work isn't ? It was more a comment to warn you that it doesn't seem to work, but maybe I did something wrong 🤷‍♂️ I still don't know how to use the secrets defined in the UI ? I didn't find an example in the documentation (each time it searches into my local secrets)
k
GOOGLE_APPLICATION_CREDENTIALS
should not be a Secret stored with us. It should be an environment variable stored in your execution environment