https://prefect.io logo
#prefect-community
Title
# prefect-community
x

xyzz

04/26/2022, 11:20 AM
would redis (https://hub.docker.com/r/ubuntu/redis) be compatible?
a

Anna Geller

04/26/2022, 12:43 PM
Probably yes, I remember someone talking about using Redis for that, let me see if I can find the relevant thread
x

xyzz

04/26/2022, 1:46 PM
so redis is on the roadmap but not ready yet... how do you use the current kv storage option?
is there any example?
a

Anna Geller

04/26/2022, 1:59 PM
no example yet, we are actively working on that - you'll likely see more updates about that in the next weeks
x

xyzz

04/28/2022, 7:35 AM
looking forward to that
all in all the experience of setting up a orion test environment that goes beyond local execution is currently pretty frustrating as the documentation lacks detail
x

xyzz

04/28/2022, 1:04 PM
I'll take a look, thanks!
šŸ‘ 1
the video shows how to set up AWS S3, but I'd like to use a local S3 storage such as minio or ceph.
Is this already supported?
would cloud need to have direct access to the storage?
a

Anna Geller

04/28/2022, 1:43 PM
what is minio or ceph?
Orion supports fsspec, so I'd say it should work:
Copy code
@sync_compatible
    @inject_client
    async def create_deployment(
        self, client: OrionClient, validate: bool = True
    ) -> UUID:
        """
        Create a deployment from the current specification.
        """
        if validate:
            await self.validate()

        flow_id = await client.create_flow(self.flow)

        # Read the flow file
        with fsspec.open(self.flow_location, "rb") as flow_file:
            flow_bytes = flow_file.read()
d

davzucky

04/28/2022, 1:53 PM
We are fspec to connect to S3 compatible storage on prem. I already extended prefect-aws to support s3 compatible storage as well. I think all the piece are
šŸ™Œ 1
@Anna Geller they are S3 compatible storage that you can use onprem or host on kubernetes Usually you need to setup the endpoint_url on boto3 client https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
x

xyzz

04/29/2022, 9:41 AM
where do you configure the endpoint url? I don't manually handle the boto3 session anywhere, its called through the storage
a

Anna Geller

04/29/2022, 7:50 PM
11 Views