Scott Zelenka
03/27/2020, 2:05 PMS3ResultHandler
doesn't expose all of the boto3.client
arguments, which prevents us from re-using this to connect to non-AWS S3 compliant Object Stores (such as MinIO).
I created a wrapper around S3ResultHandler
which simply allows the user to specify more of the parameters at initialization, and then forwards those parameters off to boto3.client
when it's initialized.
Rather than creating another Result Handler as a separate Python package, would it make more sense to modify the S3ResultHandler
in Prefect Core?
Or does it make more sense to have a dedicated Result Handler specific to MinIO, and use their Python wrapper around boto3?josh
03/27/2020, 2:39 PMs3
for the boto3.client
?Scott Zelenka
03/27/2020, 2:48 PMendpoint_url
that's not AWS. We also specify some extra config
parameters and region_name
information that's specific to MinIO instance:
s3_client = boto3.client(
service_name="s3",
region_name=self.region_name,
api_version=self.api_version,
use_ssl=self.use_ssl,
verify=self.verify,
endpoint_url=self.endpoint_url,
aws_access_key_id=aws_access_key,
aws_secret_access_key=aws_secret_access_key,
aws_session_token=None,
config=Config(**self.config)
)
initialize_client
method of S3ResultHandler
to pass through those extra parameters to boto3josh
03/27/2020, 2:49 PMAlex Cano
03/27/2020, 3:24 PMScott Zelenka
04/01/2020, 7:30 PMjosh
04/01/2020, 7:31 PM