Luke Orland
11/23/2020, 9:40 PMAWS_CREDENTIALS
secret in Prefect Cloud, should S3Result and S3Download use those credentials automatically, unless I specify something different for the boto_kwargs
argument?Kyle Moon-Wright
11/23/2020, 9:47 PMLuke Orland
11/23/2020, 9:48 PMprefect.context.get("secrets", {}).get("AWS_CREDENTIALS", {})
Kyle Moon-Wright
11/23/2020, 10:03 PMLuke Orland
11/23/2020, 10:13 PMKyle Moon-Wright
11/23/2020, 10:32 PMS3Result
is a task that will populate it’s values at runtime -so you could use the boto3_kwargs
on the S3Result which should grab those secret values from Cloud like this:
my_access_key = PrefectSecret("MY_CLOUD_SECRET_1")
my_secret_key = PrefectSecret("MY_CLOUD_SECRET_2")
my_boto3_creds = dict(
ACCESS_KEY=my_access_key,
SECRET_ACCESS_KEY=my_secret_key
)
my_result = S3Result(
#this kwarg only takes a dictionary
boto3_kwargs=my_boto3_creds
)
It has to be a dict, so you could also do:
export PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS=${AWS_CREDENTIALS}
and also pass to AWS_CREDENTIALS
to the kwarg to grab from local contextLuke Orland
11/24/2020, 3:14 AMstorage = Docker(
...,
secrets=["AWS_CREDENTIALS"],
)
Then the S3Result and S3Download objects authenticate using that secret without having to set a value for their boto3_kwargs
arguments.Saulius Beinorius
11/24/2020, 8:07 AM~/.aws
locally and using EC2 / ECS roles after deployment, that way you never have to specify the credentials in code at all 🙂