https://prefect.io logo
Join Slack
Powered by
I’m trying to figure out how to set GOOGLE_APPLICA...
# ask-community
r
I’m trying to figure out how to set GOOGLE_APPLICATIONS_CREDENTIALS via the Secrets API. I read the docs (https://docs.prefect.io/cloud/cloud_concepts/secrets.html#setting-a-secret), but only found examples of “simple” secrets (i.e. KEY = VALUE). The Google Cloud service account credentials are a .json file with many fields. How should I set the secret in this case?
c
Yea, I’d ignore the cloud concept docs for now (those are unique to our platform); these docs might be more informative for you: https://docs.prefect.io/api/unreleased/client/secrets.html TLDR; you can set secrets via env var or in 
~/.prefect/config.toml
, and for 
GOOGLE_APPLICATIONS_CREDENTIALS
specifically you’ll need to use 
json.dumps(credentials)
and strip out the escaped newlines. See this issue: https://github.com/PrefectHQ/prefect/issues/1471
👍 1
r
I added my secrets to 
~/.prefect/config.toml
according to the instructions. But when I try
Copy code
from prefect.client import Secret
Secret("GOOGLE_APPLICATION_CREDENTIALS").get()
I am getting 
ValueError: Local Secret "GOOGLE_APPLICATION_CREDENTIALS" was not found.
(I put it in the 
[context.secrets]
section)
c
did you start a new python session after you put that into config?
r
no ok that may be it
ok that worked thanks!
c
awesome, np!
r
so will that secret be sent to dask workers even if they don’t have the same config file?
c
yup actually; the secret ends up in 
context
which is explicitly shipped to the workers
r
gotcha
damn now I am getting the 
Could not deserialize key data
error when I try to actually use this. I will study https://github.com/PrefectHQ/prefect/issues/1471 more closely.
c
yea, this is an annoying consequence of toml + json I believe; on our Cloud Platform we use Vault for storing / retrieving secrets which avoids this sort of parsing issue, and we’ve got an open issue for making local secrets more pluggable so users can avoid this if they have an alternative storage mechanism: https://github.com/PrefectHQ/prefect/issues/1346
r
what I would really like to do is just set the secret interactively in my script:
Copy code
with open('pangeo-181919-0c1f01383379.json') as fp:
    token = json.load(fp)
prefect.context.secrets['GOOGLE_APPLICATION_CREDENTIALS'] = token
is that legit?
c
ah yea - you should be able to do that!
yup
totally
r
AttributeError: 'Context' object has no attribute 'secrets'
(It works if I alread have the 
config.toml
, but if I remove it, I can’t access the secrets dict)
c
ooo interesting, yea the key doesn’t populate without the config - that’s a bug! I’ll fix it ASAP. In the meantime, you can treat 
context
like a dictionary as well:
Copy code
prefect.context['secrets']['GOOGLE_APPLICATION_CREDENTIALS'] = token
r
nope 🤨
KeyError: 'secrets'
c
oh dang wrote that too quickly:
Copy code
prefect.context.setdefault('secrets', {})['GOOGLE_APPLICATION_CREDENTIALS'] = token
👍 1
r
ok that works and I like it. My recommendation would be: - Expose 
prefect.context.secrets['KEY'] = 'VALUE'
as part of the API - Add something to the docs that explains how to set a secret this way
c
yea that’s a good call out; will do!
r
(Btw, hope you don’t mind me just dropping such suggestions; I got the impression you were actively looking for user feedback.)
c
no worries, I don’t mind at all, I’m definitely interested in user feedback
and I’d love to engage more with the pangeo folks like yourself, so 💯
a
Copy code
# Authentication for Prefect Google Cloud Tasks
# <https://docs.prefect.io/api/unreleased/tasks/google.html>
json_file = open(os.environ['GOOGLE_APPLICATION_CREDENTIALS'], 'r')
context.setdefault('secrets', {})
context.secrets['GOOGLE_APPLICATION_CREDENTIALS'] = json_file.read()
json_file.close()
Open in Slack
PreviousNext
Powered by