Thanks so much for all the support and help in the...
# prefect-cloudc
Charlie Forcey
07/20/2023, 10:35 PMThanks so much for all the support and help in these channels -- so many issues are solved just by searching the dialogue. I could not find advice on this issue, so forgive me if this is a repeat. Here is the TL;DR:
• a flow is set to persist results in the RemoteFileSystem
• our S3 bucket has strict rules requiring encryption
• I cannot find a logical place to pass in , "ServerSideEncryption": "AES256" as I would on a S3 put command.
✅ 1
Charlie Forcey
07/20/2023, 10:38 PMOur flow is defined like so:
Copy code
rfs_block = RemoteFileSystem(
basepath=basepath,
settings={
"key": settings.get("aws_secret_access_id", None),
"secret": settings.get("aws_secret_access_key", None),
"client_kwargs": {"endpoint_url": settings.s3_endpoint_url},
},
)
@flow(
name="sample_flow",
persist_result=True,
result_serializer="json",
result_storage=rfs_block
)Charlie Forcey
07/20/2023, 10:39 PMTurning off encryption on the bucket shows this all working -- turning it on gives the expected permissions error since anything unencrypted is not allowed.
One happy solution is just to give up on storing results in this bucket, as we are able to save everything we need in S3 using the client directly.
Charlie Forcey
07/21/2023, 1:27 AMFrom the s3fs behind fsspec perspective, this 2017 commit is a good lead. I just have to figure out where in the block invocation (client_kwargs?) to pass this? Support for kwargs in order to support ServerSideEncryption by mariusvniekerk · Pull Request #90 · fsspec/s3fs
Charlie Forcey
07/21/2023, 2:06 AMAwesome -- the rich rewards of Prefect's decision to lean on fsspec and allow passing things along in the settings object. We needed to add this unlikely fellow from the 2017 fsspec PR #90 linked above: "s3_additional_kwargs": {"ServerSideEncryption": "AES256"} to our settings dict when defining the RemoteFileSystem.
Copy code
RemoteFileSystem(
basepath=basepath,
settings={
"key": settings.get("aws_secret_access_id", None),
"secret": settings.get("aws_secret_access_key", None),
"client_kwargs":
{
"endpoint_url": settings.s3_endpoint_url},
"s3_additional_kwargs": {"ServerSideEncryption": "AES256"}
},
)👏 2
Charlie Forcey
07/21/2023, 2:10 AMSorry to work this out in public 🙂
🙌 4
c
Chris Reuter
07/21/2023, 2:14 AMThat's what "Rubber Ducking" is all about - glad the presence of the community has helped!
👍 2
❤️ 1