https://prefect.io logo
j

Jean-Baptiste Six

01/25/2022, 2:25 PM
Hi ๐Ÿ™‚ I have an issue to use _*GCSResult, t*_he results are not stored (but there isn't any error raised). First, I tried to use environment variable :
Copy code
from dotenv import load_dotenv
load_dotenv()

@task()
def add(x, y=1):
    return x + y

with Flow("Test GCSResult", result=GCSResult(bucket='thread-prefect-flows')) as GCSResult_flow:
    first_result = add(1, y=2)
    second_result = add(x=first_result, y=100)
Without success, however :
Copy code
from dotenv import load_dotenv
load_dotenv()

def implicit():
    from google.cloud import storage

    # If you don't specify credentials when constructing the client, the
    # client library will look for credentials in the environment.
    storage_client = storage.Client()

    # Make an authenticated API request
    buckets = list(storage_client.list_buckets())
    print(buckets)

implicit()
works well, then I'm not sure to understand : "_Make sure your Prefect installation canย authenticate to Google's Cloud API_" (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult) Then, I tried to use Local Secret (https://docs.prefect.io/core/advanced_tutorials/using-results.html#example-running-a-flow-with-gcsresult)
Copy code
[context.secrets]
GOOGLE_APPLICATION_CREDENTIALS="auth/token.json"
GCP_CREDENTIALS="auth/token.json"
(don't know what to use between GCP_CREDENTIALS (prefect.utiliities.gcp) or GOOGLE_APPLICATION_CREDENTIALS (in the documentation of GCP), anyway I tried both and it didn't work neither. What did I do wrong? Please help ๐Ÿ™
a

Anna Geller

01/25/2022, 2:30 PM
1. Are you on Prefect Cloud or Server? 2. What agent do you use? GCP credentials must be available on your agent
if you SSH to your agentโ€™s machine and type this in your terminal, does it work? This is how you can test whether your agent has the right permissions:
Copy code
gsutil ls -r <gs://thread-prefect-flows>
j

Jean-Baptiste Six

01/25/2022, 2:34 PM
Copy code
gsutil ls -r <gs://thread-prefect-flows>
works well
a

Anna Geller

01/25/2022, 2:35 PM
you can also set it when you start your agent:
Copy code
prefect agent xxx start --env PREFECT__CONTEXT__SECRETS__GCP_CREDENTIALS="your_creds"
j

Jean-Baptiste Six

01/25/2022, 2:36 PM
For the first question, I use Prefect Cloud, but I want to make it work in local For the second one, i don't know, what are the different kind of agents ?
I'm using a local agent. To run the flow, I just do :
Copy code
def test_GCSResult():
    GCSResult_flow.run()
a

Anna Geller

01/25/2022, 2:44 PM
to use results, you need to deploy your flow to a backend, i.e. you would need to register your flow and trigger it e.g. from the UI/CLI/API.
j

Jean-Baptiste Six

01/25/2022, 2:48 PM
I see ! It is not possible to test GCSResult in local then ?
a

Anna Geller

01/25/2022, 2:51 PM
It depends on what do you consider local ๐Ÿ™‚ you can use it with your local agent and then effectively everything runs locally, but the execution (incl. result handling) is governed by the Cloud API
๐Ÿ™ 1
j

Jean-Baptiste Six

01/25/2022, 2:53 PM
@Luis Arias FYI
@Anna Geller I have investigated, and I have the following error :
Copy code
Unexpected error: AttributeError("'str' object has no attribute 'keys'")
Right now I have the following secret :
Copy code
GCP_CREDENTIALS = "auth/token.json"
where is stored my credentials for GCP. But when I see the error, it make me think that I need to directly give the content of the json file in the secret, is that correct ?
k

Kevin Kho

01/25/2022, 6:58 PM
So if you use
GOOGLE_APPLICATION_CREDENTIALS
, you point it to the json. If you use the
GCP_CREDENTIALS
, this is a secret that contains the content of the JSON
GOOGLE_APPLICATION_CREDENTIALS
is google convention.
GCP_CREDENTIALS
is a Prefect convention because we would store the content of the JSON in Cloud
๐Ÿ™ 1
If you use
GOOGLE_APPLICATION_CREDENTIALS
, you just need the
.json
file in the execution environment
j

Jean-Baptiste Six

01/26/2022, 8:49 AM
Thanks @Kevin Kho!
How could I use the secrets defined in the UI ?
Btw, I succeeded to make it work with GCP_CREDENTIALS (by converting the json into TOML file into ~/.prefect/config.toml (not super intuitive)) But the secret GOOGLE_APPLICATION_CREDENTIALS = "auth/token.json" doesn't seem to work, is it a bug ?
a

Anna Geller

01/26/2022, 10:59 AM
As Kevin explained, GCP_CREDENTIALS is the Prefect-specific implementation so that you can store the secret in the Cloud Secret backend. If that works for you, no need to worry about GOOGLE_APPLICATION_CREDENTIALS which are intended to only reference the path to the JSON file describing permissions of the service account.
j

Jean-Baptiste Six

01/26/2022, 2:14 PM
I know ๐Ÿ‘ but if GCP_CREDENTIALS is not defined, GOOGLE_APPLICATION_CREDENTIALS should work isn't ? It was more a comment to warn you that it doesn't seem to work, but maybe I did something wrong ๐Ÿคทโ€โ™‚๏ธ I still don't know how to use the secrets defined in the UI ? I didn't find an example in the documentation (each time it searches into my local secrets)
k

Kevin Kho

01/26/2022, 2:39 PM
GOOGLE_APPLICATION_CREDENTIALS
should not be a Secret stored with us. It should be an environment variable stored in your execution environment