Scott Zelenka

    Scott Zelenka

    2 years ago
    Hi Team, I'm looking for guidance on the correct approach towards modifying a result_handler. Specifically, the
    S3ResultHandler
    doesn't expose all of the
    boto3.client
    arguments, which prevents us from re-using this to connect to non-AWS S3 compliant Object Stores (such as MinIO). I created a wrapper around
    S3ResultHandler
    which simply allows the user to specify more of the parameters at initialization, and then forwards those parameters off to
    boto3.client
    when it's initialized. Rather than creating another Result Handler as a separate Python package, would it make more sense to modify the
    S3ResultHandler
    in Prefect Core? Or does it make more sense to have a dedicated Result Handler specific to MinIO, and use their Python wrapper around boto3?
    j

    josh

    2 years ago
    For this are you still specifying
    s3
    for the
    boto3.client
    ?
    Scott Zelenka

    Scott Zelenka

    2 years ago
    yes, it's just that we'd need to specify a specific
    endpoint_url
    that's not AWS. We also specify some extra
    config
    parameters and
    region_name
    information that's specific to MinIO instance:
    s3_client = boto3.client(
                service_name="s3",
                region_name=self.region_name,
                api_version=self.api_version,
                use_ssl=self.use_ssl,
                verify=self.verify,
                endpoint_url=self.endpoint_url,
                aws_access_key_id=aws_access_key,
                aws_secret_access_key=aws_secret_access_key,
                aws_session_token=None,
                config=Config(**self.config)
            )
    that's really the only change, is that it overloaded the
    initialize_client
    method of
    S3ResultHandler
    to pass through those extra parameters to boto3
    j

    josh

    2 years ago
    Awesome! Yeah we should definitely expand the current S3ResultHandler to accept all of the boto3 client kwargs
    a

    Alex Cano

    2 years ago
    We should also remember to include this in the migration from result handlers -> result objects, just to make sure it doesn’t get lost in the madness!
    Scott Zelenka

    Scott Zelenka

    2 years ago
    I don't know if I did this correctly, I made a GitHub Issue here https://github.com/PrefectHQ/prefect/issues/2240 and submitted a PR here https://github.com/PrefectHQ/prefect/pull/2241 Is there anything else I need to do to link the PR to the Issue?
    j

    josh

    2 years ago
    Nope! Usually we say something like “Closes #2240” in the description so GitHub auto-closes the issue but it isn’t absolutely necessary
    Thanks for making the PR and outlining the issue!