https://prefect.io logo
Powered by
Title
r

Ryan Abernathey

09/09/2019, 7:42 PM
I’m trying to figure out how to set GOOGLE_APPLICATIONS_CREDENTIALS via the Secrets API. I read the docs (https://docs.prefect.io/cloud/cloud_concepts/secrets.html#setting-a-secret), but only found examples of “simple” secrets (i.e. KEY = VALUE). The Google Cloud service account credentials are a .json file with many fields. How should I set the secret in this case?
c

Chris White

09/09/2019, 7:57 PM
Yea, I’d ignore the cloud concept docs for now (those are unique to our platform); these docs might be more informative for you: https://docs.prefect.io/api/unreleased/client/secrets.html TLDR; you can set secrets via env var or in 
~/.prefect/config.toml
, and for 
GOOGLE_APPLICATIONS_CREDENTIALS
specifically you’ll need to use 
json.dumps(credentials)
and strip out the escaped newlines. See this issue: https://github.com/PrefectHQ/prefect/issues/1471
👍 1
r

Ryan Abernathey

09/09/2019, 8:19 PM
I added my secrets to 
~/.prefect/config.toml
according to the instructions. But when I try 
from prefect.client import Secret
Secret("GOOGLE_APPLICATION_CREDENTIALS").get()
I am getting 
ValueError: Local Secret "GOOGLE_APPLICATION_CREDENTIALS" was not found.
(I put it in the 
[context.secrets]
section)
c

Chris White

09/09/2019, 8:20 PM
did you start a new python session after you put that into config?
r

Ryan Abernathey

09/09/2019, 8:21 PM
no ok that may be it
ok that worked thanks!
c

Chris White

09/09/2019, 8:21 PM
awesome, np!
r

Ryan Abernathey

09/09/2019, 8:21 PM
so will that secret be sent to dask workers even if they don’t have the same config file?
c

Chris White

09/09/2019, 8:22 PM
yup actually; the secret ends up in 
context
which is explicitly shipped to the workers
r

Ryan Abernathey

09/09/2019, 8:22 PM
gotcha
damn now I am getting the 
Could not deserialize key data
error when I try to actually use this. I will study https://github.com/PrefectHQ/prefect/issues/1471 more closely.
c

Chris White

09/09/2019, 8:25 PM
yea, this is an annoying consequence of toml + json I believe; on our Cloud Platform we use Vault for storing / retrieving secrets which avoids this sort of parsing issue, and we’ve got an open issue for making local secrets more pluggable so users can avoid this if they have an alternative storage mechanism: https://github.com/PrefectHQ/prefect/issues/1346
r

Ryan Abernathey

09/09/2019, 8:26 PM
what I would really like to do is just set the secret interactively in my script: 
with open('pangeo-181919-0c1f01383379.json') as fp:
    token = json.load(fp)
prefect.context.secrets['GOOGLE_APPLICATION_CREDENTIALS'] = token
is that legit?
c

Chris White

09/09/2019, 8:27 PM
ah yea - you should be able to do that!
yup
totally
r

Ryan Abernathey

09/09/2019, 8:27 PM
AttributeError: 'Context' object has no attribute 'secrets'
(It works if I alread have the 
config.toml
, but if I remove it, I can’t access the secrets dict)
c

Chris White

09/09/2019, 8:28 PM
ooo interesting, yea the key doesn’t populate without the config - that’s a bug! I’ll fix it ASAP. In the meantime, you can treat 
context
like a dictionary as well: 
prefect.context['secrets']['GOOGLE_APPLICATION_CREDENTIALS'] = token
r

Ryan Abernathey

09/09/2019, 8:30 PM
nope 🤨
KeyError: 'secrets'
c

Chris White

09/09/2019, 8:33 PM
oh dang wrote that too quickly: 
prefect.context.setdefault('secrets', {})['GOOGLE_APPLICATION_CREDENTIALS'] = token
👍 1
r

Ryan Abernathey

09/09/2019, 8:35 PM
ok that works and I like it. My recommendation would be: - Expose 
prefect.context.secrets['KEY'] = 'VALUE'
as part of the API - Add something to the docs that explains how to set a secret this way
c

Chris White

09/09/2019, 8:36 PM
yea that’s a good call out; will do!
r

Ryan Abernathey

09/09/2019, 8:37 PM
(Btw, hope you don’t mind me just dropping such suggestions; I got the impression you were actively looking for user feedback.)
c

Chris White

09/09/2019, 8:38 PM
no worries, I don’t mind at all, I’m definitely interested in user feedback
and I’d love to engage more with the pangeo folks like yourself, so 💯
a

Abraão Zaidan

11/08/2019, 8:16 PM
# Authentication for Prefect Google Cloud Tasks
# <https://docs.prefect.io/api/unreleased/tasks/google.html>
json_file = open(os.environ['GOOGLE_APPLICATION_CREDENTIALS'], 'r')
context.setdefault('secrets', {})
context.secrets['GOOGLE_APPLICATION_CREDENTIALS'] = json_file.read()
json_file.close()
#prefect-communityJoin Slack
Powered by