https://prefect.io logo
Title
d

Darragh

06/10/2020, 9:03 PM
Back again, looking for advice on secrets. I’m trying to inject secrets into Fargate containers and not getting very far. As I understand it, the native AWS Secret Manager support allows me to specify a secret ARN, and if my secret name is something like CREDS, I add the following to the FargateAgent config:
"secrets":[{"name": "CREDS", "valueFrom": "arn:aws:secretsmanager:eu-west-1:11111111:secret:local/aws/credentials-abcd"}]
In Flow, I read like this:
creds = prefect.context.secrets.CREDS
But I keep getting the following:
AttributeError: 'dict' object has no attribute 'CREDS'
Confused face.
c

Chris White

06/10/2020, 9:04 PM
Hey Darragh - I don’t fully understand the situation but could you try
creds = prefect.context.secrets["CREDS"]
?
d

Darragh

06/10/2020, 9:09 PM
Hey Chris, tired that there, now getting
KeyError: 'CREDS'
I can see in the TaskDefinition that the agent is trying to inject the secret as an environment variabe with key
CREDS
and value
arn:aws:secretsmanager:eu-west-1:11111111:/local/aws/credentials-abcd
@Chris White Correction, the proper error is
Unexpected error: KeyError('CREDS',)
👍 1
c

Chris White

06/10/2020, 9:16 PM
and you are setting the
PREFECT__CONTEXT__SECRETS__CREDS
environment variable?
d

Darragh

06/10/2020, 9:19 PM
No I have the CREDS item in SecretManager - from the FargateAgent docs it looked like all I needed was the secret in SecretsManger, passing it as config to FargateAgent like so :
"secrets":[""name": "CREDS", "valueFrom": "arn:aws:secretsmanager:eu-west-1:11111111local/aws/credentials-abcd"}]
and then reading it in the flow as
prefect.context.secrets["CREDS"]
I didn’t spot anything about the format of environment variable you mentioned above?
c

Chris White

06/10/2020, 9:20 PM
sorry I’m not as familiar with the fargate agent can you show me what docs you’re referring to? If you set the env var that I specified above then that will definitely set the secret in the location you’re referencing
d

Darragh

06/10/2020, 9:22 PM
This line
This adds support for Native AWS Secrets Manager and/or Parameter Store in your flows.
in https://docs.prefect.io/orchestration/agents/fargate.html So I need the env var as well, as part of the agent config? Do I still need to add it in the secrets section of the config?
I’m not quite getting what you’re saying with the env var - Am I setting that to be
export PREFECT__CONTEXT__SECRETS__CREDS=arn:aws:secrets....
in the env where i start the agent? Do I need to pass it as an variable in the
"environments"
section as well?
c

Chris White

06/10/2020, 9:31 PM
So there are two concepts at play here: AWS secrets and the Prefect Secrets interface. I’m not 100% sure what your goal is but the doc you referenced is how to place AWS secrets into your Flow’s runtime environment. Independently of AWS, you can set secrets for the Prefect interface using the environment variable approach I mentioned above. If your goal is to use the Prefect Secrets interface backed by AWS secrets manager, I highly recommend you subclass the base SecretTask class and interface with AWS that way (there’s actually an open issue for this that you can contribute to if you’d like! https://github.com/PrefectHQ/prefect/issues/2069)
d

Darragh

06/10/2020, 9:37 PM
Thanks for that Chris, looking forward to seeing that make it in! Main case is that I have a bunch of secret data I need access to. It’ll be stored in AWS Secrets rather than as env vars, because they’ll be more secure that way. If I just want to put AWS secrets in the Flow runtime, am I right in what I was asking above? I create my secret in AWS, add it to the “secrets” config section of the Agent, and then read it in the Flow by some method I still don’t get?
Or am I not making it simple enough - by adding it in “secrets”, can I then just treat it as an env variable in the flow?
Yeah that’s what it was - so the flow seems to be: • Create secret in AWS Secrets Manager - Copy ARN • Add Secret Name + ARN into the “secrets” config section of your FargateAgent • Read as normal env variable in Flow - Prefect subs the secret value out for the ARN
:upvote: 1
Thanks for your help @Chris White!
c

Chris White

06/10/2020, 10:56 PM
Sorry for going radio silent I had to pop into a meeting but I’m really glad you figured it out!
@Marvin archive “How to set and use AWS secrets within my Flow running on a Fargate Agent”