r

    Ryan Abernathey

    3 years ago
    I’m trying to figure out how to set GOOGLE_APPLICATIONS_CREDENTIALS via the Secrets API. I read the docs (https://docs.prefect.io/cloud/cloud_concepts/secrets.html#setting-a-secret), but only found examples of “simple” secrets (i.e. KEY = VALUE). The Google Cloud service account credentials are a .json file with many fields. How should I set the secret in this case?
    Chris White

    Chris White

    3 years ago
    Yea, I’d ignore the cloud concept docs for now (those are unique to our platform); these docs might be more informative for you: https://docs.prefect.io/api/unreleased/client/secrets.html TLDR; you can set secrets via env var or in
    ~/.prefect/config.toml
    , and for
    GOOGLE_APPLICATIONS_CREDENTIALS
    specifically you’ll need to use
    json.dumps(credentials)
    and strip out the escaped newlines. See this issue: https://github.com/PrefectHQ/prefect/issues/1471
    r

    Ryan Abernathey

    3 years ago
    I added my secrets to
    ~/.prefect/config.toml
    according to the instructions. But when I try
    from prefect.client import Secret
    Secret("GOOGLE_APPLICATION_CREDENTIALS").get()
    I am getting
    ValueError: Local Secret "GOOGLE_APPLICATION_CREDENTIALS" was not found.
    (I put it in the
    [context.secrets]
    section)
    Chris White

    Chris White

    3 years ago
    did you start a new python session after you put that into config?
    r

    Ryan Abernathey

    3 years ago
    no ok that may be it
    ok that worked thanks!
    Chris White

    Chris White

    3 years ago
    awesome, np!
    r

    Ryan Abernathey

    3 years ago
    so will that secret be sent to dask workers even if they don’t have the same config file?
    Chris White

    Chris White

    3 years ago
    yup actually; the secret ends up in
    context
    which is explicitly shipped to the workers
    r

    Ryan Abernathey

    3 years ago
    gotcha
    damn now I am getting the
    Could not deserialize key data
    error when I try to actually use this. I will study https://github.com/PrefectHQ/prefect/issues/1471 more closely.
    Chris White

    Chris White

    3 years ago
    yea, this is an annoying consequence of toml + json I believe; on our Cloud Platform we use Vault for storing / retrieving secrets which avoids this sort of parsing issue, and we’ve got an open issue for making local secrets more pluggable so users can avoid this if they have an alternative storage mechanism: https://github.com/PrefectHQ/prefect/issues/1346
    r

    Ryan Abernathey

    3 years ago
    what I would really like to do is just set the secret interactively in my script:
    with open('pangeo-181919-0c1f01383379.json') as fp:
        token = json.load(fp)
    prefect.context.secrets['GOOGLE_APPLICATION_CREDENTIALS'] = token
    is that legit?
    Chris White

    Chris White

    3 years ago
    ah yea - you should be able to do that!
    yup
    totally
    r

    Ryan Abernathey

    3 years ago
    AttributeError: 'Context' object has no attribute 'secrets'
    (It works if I alread have the
    config.toml
    , but if I remove it, I can’t access the secrets dict)
    Chris White

    Chris White

    3 years ago
    ooo interesting, yea the key doesn’t populate without the config - that’s a bug! I’ll fix it ASAP. In the meantime, you can treat
    context
    like a dictionary as well:
    prefect.context['secrets']['GOOGLE_APPLICATION_CREDENTIALS'] = token
    r

    Ryan Abernathey

    3 years ago
    nope 🤨
    KeyError: 'secrets'
    Chris White

    Chris White

    3 years ago
    oh dang wrote that too quickly:
    prefect.context.setdefault('secrets', {})['GOOGLE_APPLICATION_CREDENTIALS'] = token
    r

    Ryan Abernathey

    3 years ago
    ok that works and I like it. My recommendation would be:- Expose
    prefect.context.secrets['KEY'] = 'VALUE'
    as part of the API- Add something to the docs that explains how to set a secret this way
    Chris White

    Chris White

    3 years ago
    yea that’s a good call out; will do!
    r

    Ryan Abernathey

    3 years ago
    (Btw, hope you don’t mind me just dropping such suggestions; I got the impression you were actively looking for user feedback.)
    Chris White

    Chris White

    3 years ago
    no worries, I don’t mind at all, I’m definitely interested in user feedback
    and I’d love to engage more with the pangeo folks like yourself, so 💯
    a

    Abraão Zaidan

    2 years ago
    # Authentication for Prefect Google Cloud Tasks
    # <https://docs.prefect.io/api/unreleased/tasks/google.html>
    json_file = open(os.environ['GOOGLE_APPLICATION_CREDENTIALS'], 'r')
    context.setdefault('secrets', {})
    context.secrets['GOOGLE_APPLICATION_CREDENTIALS'] = json_file.read()
    json_file.close()