Robin
07/15/2020, 5:11 PMAWS secrets manager
with prefect?
1️⃣ Fetch the credentials within each task
2️⃣ Create a task that fetches the credentials once
3️⃣ another option
Which one is considered most save?
We are aware of Prefect secrets, but would also like to save all secrets at one place, e.g. AWS secrets manager…Sven Teresniak
07/15/2020, 5:12 PMRobin
07/15/2020, 5:15 PMimport boto3
secret_name = "secretname"
region_name = "eu-central-1"
# Create a Secrets Manager client
session = boto3.session.Session()
client = session.client(service_name="secretsmanager", region_name=region_name,)
get_secret_value_response = client.get_secret_value(SecretId=secret_name)
print(get_secret_value_response["SecretString"])
Thus, still relying on either AWS access key and secret key in env variables or on some temporary access key, secret key and session token generation in a VPC …. 🤔
We did not test the second option though.I have the same problem here: how to use s3 credentials and not putting them in $ENV everywhere…Yeah, we are still thinking about how to best manage secrets in general and also ask ourself that question 🤔
josh
07/15/2020, 5:20 PMPrefectSecret
task which uses Prefect secrets is a simple task that subclasses the SecretBase
class. So if you wanted to make your own AWS secrets manager secret task you would do something like this:
class AWSSecretManagerTask(SecretBase):
def __init__(self, name, **kwargs):
# could initialize client here
# or during run
kwargs["name"] = name
super().__init__(**kwargs)
@defaults_from_attrs("name")
def run(self, name: str = None):
# use boto here to grab and
# return secret
And then that could be used in a flow:
with Flow() as f:
s = AWSSecretManagerTask('secret')
task_that_uses_secret(s)
Chris White
07/15/2020, 5:23 PMRobin
07/15/2020, 6:11 PMChris White
07/15/2020, 6:12 PMRobin
07/15/2020, 6:39 PM