Hey guys, looking for advice on best practice with Secrets for our use case. we’re using an EC2 instance to host our Prefect Server, and the main execution env is Fargate. Fargate tasks will need to be able to access various secrets in order to do stuff like S3Download,/Upload, various other things. From reading the docs on the various ways to inject a Secret into the system this is what it currently seems like… For static items like AWS creds • Configure the EC2 env with variables that can be interpolated into the config.toml • Start Fargate agent with this same env, configure the “secret”s section of the agent with the secrets to inject • Flows on Fargate using S3Tasks can pull those injected vars from the environment and run This will work for a handful of cases but not all, so I’m trying to figure the best method of getting runtime variables/secrets into a Flow. A given Flow can read from prefect.context.secrets… but how do I actually propagate them through the system above without doing it manually/adding them to the Core UI?

Hi @Darragh, good question. Do you mind describing the cases in which retrieving the secret as part of a task won't work? It's also worth shouting out that we manage remote secrets for you as part of Prefect Cloud. The Scheduler tier is free if you'd like to give it a whirl!

Hey @Zachary Hughes, sorry I should have been clearer - retrieving the secret in a flow should be reasonably handy, using the prefect.context.secret stuff. My gap is how to get it from the server, to the agent, to the flow? And using prefect cloud is sadly not an option for us 😆

Okay, gotcha! Given your situation, your best bet might be to bypass the server -> agent -> flow path and pull your secrets directly from a central store of your choice as part of your flow. With that value retrieved, you'd be able to use the value of your secret however you'd like.

interesting - do you know of any cases where users have pulled from an AWS store?

I think we've had one or two folks create custom secret classes pulling from Google's secret manager, so I'm betting pulling from AWS's secret manager would work as well.

Yeah that's the next plan 😁👍 thanks Zachary!