Back again, looking for advice on secrets. I’m try...
# prefect-community
Back again, looking for advice on secrets. I’m trying to inject secrets into Fargate containers and not getting very far. As I understand it, the native AWS Secret Manager support allows me to specify a secret ARN, and if my secret name is something like CREDS, I add the following to the FargateAgent config:
"secrets":[{"name": "CREDS", "valueFrom": "arn:aws:secretsmanager:eu-west-1:11111111:secret:local/aws/credentials-abcd"}]
In Flow, I read like this:
creds = prefect.context.secrets.CREDS
But I keep getting the following:
AttributeError: 'dict' object has no attribute 'CREDS'
Confused face.
Hey Darragh - I don’t fully understand the situation but could you try
Copy code
creds = prefect.context.secrets["CREDS"]
Hey Chris, tired that there, now getting
KeyError: 'CREDS'
I can see in the TaskDefinition that the agent is trying to inject the secret as an environment variabe with key
and value
@Chris White Correction, the proper error is
Unexpected error: KeyError('CREDS',)
👍 1
and you are setting the
environment variable?
No I have the CREDS item in SecretManager - from the FargateAgent docs it looked like all I needed was the secret in SecretsManger, passing it as config to FargateAgent like so :
"secrets":[""name": "CREDS", "valueFrom": "arn:aws:secretsmanager:eu-west-1:11111111local/aws/credentials-abcd"}]
and then reading it in the flow as
I didn’t spot anything about the format of environment variable you mentioned above?
sorry I’m not as familiar with the fargate agent can you show me what docs you’re referring to? If you set the env var that I specified above then that will definitely set the secret in the location you’re referencing
This line
This adds support for Native AWS Secrets Manager and/or Parameter Store in your flows.
in So I need the env var as well, as part of the agent config? Do I still need to add it in the secrets section of the config?
I’m not quite getting what you’re saying with the env var - Am I setting that to be
export PREFECT__CONTEXT__SECRETS__CREDS=arn:aws:secrets....
in the env where i start the agent? Do I need to pass it as an variable in the
section as well?
So there are two concepts at play here: AWS secrets and the Prefect Secrets interface. I’m not 100% sure what your goal is but the doc you referenced is how to place AWS secrets into your Flow’s runtime environment. Independently of AWS, you can set secrets for the Prefect interface using the environment variable approach I mentioned above. If your goal is to use the Prefect Secrets interface backed by AWS secrets manager, I highly recommend you subclass the base SecretTask class and interface with AWS that way (there’s actually an open issue for this that you can contribute to if you’d like!
Thanks for that Chris, looking forward to seeing that make it in! Main case is that I have a bunch of secret data I need access to. It’ll be stored in AWS Secrets rather than as env vars, because they’ll be more secure that way. If I just want to put AWS secrets in the Flow runtime, am I right in what I was asking above? I create my secret in AWS, add it to the “secrets” config section of the Agent, and then read it in the Flow by some method I still don’t get?
Or am I not making it simple enough - by adding it in “secrets”, can I then just treat it as an env variable in the flow?
Yeah that’s what it was - so the flow seems to be: • Create secret in AWS Secrets Manager - Copy ARN • Add Secret Name + ARN into the “secrets” config section of your FargateAgent • Read as normal env variable in Flow - Prefect subs the secret value out for the ARN
upvote 1
Thanks for your help @Chris White!
Sorry for going radio silent I had to pop into a meeting but I’m really glad you figured it out!
@Marvin archive “How to set and use AWS secrets within my Flow running on a Fargate Agent”