https://prefect.io logo
Title
r

Robin

07/15/2020, 5:11 PM
Dear all, how to best use
AWS secrets manager
with prefect? 1️⃣ Fetch the credentials within each task 2️⃣ Create a task that fetches the credentials once 3️⃣ another option Which one is considered most save? We are aware of Prefect secrets, but would also like to save all secrets at one place, e.g. AWS secrets manager…
2️⃣ 3
s

Sven Teresniak

07/15/2020, 5:12 PM
How do you fetch credentials?
I have the same problem here: how to use s3 credentials and not putting them in $ENV everywhere…
r

Robin

07/15/2020, 5:15 PM
import boto3

secret_name = "secretname"
region_name = "eu-central-1"

# Create a Secrets Manager client
session = boto3.session.Session()
client = session.client(service_name="secretsmanager", region_name=region_name,)

get_secret_value_response = client.get_secret_value(SecretId=secret_name)

print(get_secret_value_response["SecretString"])
Thus, still relying on either AWS access key and secret key in env variables or on some temporary access key, secret key and session token generation in a VPC …. 🤔 We did not test the second option though.
I have the same problem here: how to use s3 credentials and not putting them in $ENV everywhere…
Yeah, we are still thinking about how to best manage secrets in general and also ask ourself that question 🤔
j

josh

07/15/2020, 5:20 PM
Food for thought: the default
PrefectSecret
task which uses Prefect secrets is a simple task that subclasses the
SecretBase
class. So if you wanted to make your own AWS secrets manager secret task you would do something like this:
class AWSSecretManagerTask(SecretBase):
    def __init__(self, name, **kwargs):
        # could initialize client here
        # or during run
        kwargs["name"] = name
        super().__init__(**kwargs)

    @defaults_from_attrs("name")
    def run(self, name: str = None):
        # use boto here to grab and 
        # return secret
And then that could be used in a flow:
with Flow() as f:
    s = AWSSecretManagerTask('secret')
    task_that_uses_secret(s)
🚀 1
c

Chris White

07/15/2020, 5:23 PM
r

Robin

07/15/2020, 6:11 PM
Thank you two! I found the link to #2069 but that was not so straight-forward to understand for me. Having played a bit with boto and with the code snippet from @josh, I can think of putting these two pieces together and closing that PR, if that’s appreciated? 🙂
c

Chris White

07/15/2020, 6:12 PM
That would be very much appreciated! Feel free to open a draft / WIP PR as well and we can collaborate with you if you have any questions
⬆️ 1
r

Robin

07/15/2020, 6:39 PM
Alright! Am currently experimenting a bit with it and will open a PR soonish 🙂
For the sake of completeness: Here the link to the related Pull Request 🙂
💯 1