davzucky

    davzucky

    2 months ago
    S3StorageBlock is not working with Minio as we cannot specify the endpoint_url. This is something I added on the collection-aws. Will this block move to the collection-aws ? If not, how can I do a push request to the branch ?
    Anna Geller

    Anna Geller

    2 months ago
    great question! as part of the latest release, storage blocks, including default storage and storage CLI were removed in favor of filesystems - you would need to use a RemoteFileSystem with your minio S3 endpoint instead
    syntax:
    from prefect import task, flow
    from prefect import get_run_logger
    from prefect.deployments import Deployment
    from prefect.packaging import FilePackager
    from prefect.filesystems import RemoteFileSystem
    
    
    @task
    def say_hi(user_name: str):
        logger = get_run_logger()
        <http://logger.info|logger.info>("Hello %s!", user_name)
    
    
    @flow
    def hello(user: str = "world"):
        say_hi(user)
    
    
    Deployment(
        flow=hello,
        name="s3_file_packager_with_remote_s3fs",
        packager=FilePackager(
            filesystem=RemoteFileSystem(basepath="<s3://prefect-orion/flows/>")
        ),
    )
    if you can't set up your endpoint properly with settings on RemoteFileSystem, please LMK and I'll open an issue
    davzucky

    davzucky

    2 months ago
    Good point. Forgot about that one using fsspec. Let me try it