https://prefect.io logo
Title
s

Scott Zelenka

07/01/2020, 3:06 PM
in order to use the new
S3Result
instead of
S3ResultHandler
, do we need to have a PrefectSecret named
AWS_CREDENTIALS
? Previously with
S3ResultHandler
there was a kwarg for `aws_credentials_secret`(which allowed us to specify different credentials for different Flows), but that doesn't seem to be present on the new
S3Result
. Trying to figure out how to pass those credentials to
S3Result
j

josh

07/01/2020, 3:17 PM
Hey @Scott Zelenka using a secret called
AWS_CREDENTIALS
would be your best bet https://docs.prefect.io/core/concepts/secrets.html#default-secrets as the
S3Result
does not accept credentials directly. The
S3Result
does not currently allow for passing in credentials directly to the result however the client it uses has an option for it https://github.com/PrefectHQ/prefect/blob/master/src/prefect/utilities/aws.py so we could expose this on the Result class
Actually I overlooked. The
S3Result
has an option for
boto3_kwargs
which you can set boto3 credentials in if you choose
So you could do something like:
S3Result(boto3_kwargs={"aws_access_key_id": "...", etc.})
s

Scott Zelenka

07/01/2020, 3:45 PM
we have a few Agents in different environments. And have the results_handler (or results now) configured at the Flow context. The Agents that are on-prem, are configured to persist their results to an internal Mino server (via S3ResultHandler with a PrefectSecret defined). We're exploring migrating some of the Flows to AWS, where they'd be writing to AWS's S3 (with different credentials). Since both of them would be in the same Prefect Cloud Tenant, we couldn't re-use the
AWS_CREDENTIALS
secret for both use cases.
I don't know if we can use the boto3_kwargs to pass the aws_access_key_id, etc. because those are stored in a PrefectSecret
Or maybe this will just help accelerate our migration to AWS and off of MinIO đŸ™‚
j

josh

07/01/2020, 3:49 PM
Haha that’s one way of looking at it! IMO I think we could expose the name of the secret to be used by the AWS
get_boto_client
utility and still have it use a default of
AWS_CREDENTIALS
s

Scott Zelenka

07/01/2020, 3:50 PM
I'll open a feature request in GitHub for that
j

josh

07/01/2020, 3:50 PM
Awesome thanks @Scott Zelenka I think it should be straightforward to add