https://prefect.io logo
c

Charlie Forcey

07/20/2023, 10:35 PM
Thanks so much for all the support and help in these channels -- so many issues are solved just by searching the dialogue. I could not find advice on this issue, so forgive me if this is a repeat. Here is the TL;DR: • a flow is set to persist results in the RemoteFileSystem • our S3 bucket has strict rules requiring encryption • I cannot find a logical place to pass in , "ServerSideEncryption": "AES256" as I would on a S3 put command.
1
Our flow is defined like so:
Copy code
rfs_block = RemoteFileSystem(
            basepath=basepath,
            settings={
                "key": settings.get("aws_secret_access_id", None),
                "secret": settings.get("aws_secret_access_key", None),
                "client_kwargs": {"endpoint_url": settings.s3_endpoint_url},
            },
        )

@flow(
    name="sample_flow",
    persist_result=True,
    result_serializer="json",
    result_storage=rfs_block
)
Turning off encryption on the bucket shows this all working -- turning it on gives the expected permissions error since anything unencrypted is not allowed. One happy solution is just to give up on storing results in this bucket, as we are able to save everything we need in S3 using the client directly.
From the s3fs behind fsspec perspective, this 2017 commit is a good lead. I just have to figure out where in the block invocation (client_kwargs?) to pass this? Support for kwargs in order to support ServerSideEncryption by mariusvniekerk · Pull Request #90 · fsspec/s3fs
Awesome -- the rich rewards of Prefect's decision to lean on fsspec and allow passing things along in the settings object. We needed to add this unlikely fellow from the 2017 fsspec PR #90 linked above: "s3_additional_kwargs": {"ServerSideEncryption": "AES256"} to our settings dict when defining the RemoteFileSystem.
Copy code
RemoteFileSystem(
  basepath=basepath,
  settings={
    "key": settings.get("aws_secret_access_id", None),
    "secret": settings.get("aws_secret_access_key", None),
    "client_kwargs": 
      {
        "endpoint_url": settings.s3_endpoint_url},
        "s3_additional_kwargs": {"ServerSideEncryption": "AES256"}
      },
)
👏 2
Sorry to work this out in public 🙂
🙌 4
c

Chris Reuter

07/21/2023, 2:14 AM
That's what "Rubber Ducking" is all about - glad the presence of the community has helped!
👍 2
❤️ 1