Dear all, how to best use

AWS secrets manager

with prefect? 1️⃣ Fetch the credentials within each task 2️⃣ Create a task that fetches the credentials once 3️⃣ another option Which one is considered most save? We are aware of Prefect secrets, but would also like to save all secrets at one place, e.g. AWS secrets manager…

How do you fetch credentials?

import boto3

secret_name = "secretname"
region_name = "eu-central-1"

# Create a Secrets Manager client
session = boto3.session.Session()
client = session.client(service_name="secretsmanager", region_name=region_name,)

get_secret_value_response = client.get_secret_value(SecretId=secret_name)

print(get_secret_value_response["SecretString"])

Thus, still relying on either AWS access key and secret key in env variables or on some temporary access key, secret key and session token generation in a VPC …. 🤔 We did not test the second option though.

Food for thought: the default

PrefectSecret

task which uses Prefect secrets is a simple task that subclasses the

SecretBase

class. So if you wanted to make your own AWS secrets manager secret task you would do something like this:

class AWSSecretManagerTask(SecretBase):
    def __init__(self, name, **kwargs):
        # could initialize client here
        # or during run
        kwargs["name"] = name
        super().__init__(**kwargs)

    @defaults_from_attrs("name")
    def run(self, name: str = None):
        # use boto here to grab and 
        # return secret

And then that could be used in a flow:

with Flow() as f:
    s = AWSSecretManagerTask('secret')
    task_that_uses_secret(s)

adding onto that: https://github.com/PrefectHQ/prefect/issues/2069 😉

Thank you two! I found the link to #2069 but that was not so straight-forward to understand for me. Having played a bit with boto and with the code snippet from @josh, I can think of putting these two pieces together and closing that PR, if that’s appreciated? 🙂

That would be very much appreciated! Feel free to open a draft / WIP PR as well and we can collaborate with you if you have any questions

Alright! Am currently experimenting a bit with it and will open a PR soonish 🙂