S3StorageBlock is not working with Minio as we cannot specify the endpoint_url. This is something I added on the collection-aws.
Will this block move to the collection-aws ? If not, how can I do a push request to the branch ?
✅ 1
a
Anna Geller
07/22/2022, 9:40 AM
great question! as part of the latest release, storage blocks, including default storage and storage CLI were removed in favor of filesystems - you would need to use a RemoteFileSystem with your minio S3 endpoint instead
Anna Geller
07/22/2022, 9:45 AM
syntax:
Copy code
from prefect import task, flow
from prefect import get_run_logger
from prefect.deployments import Deployment
from prefect.packaging import FilePackager
from prefect.filesystems import RemoteFileSystem
@task
def say_hi(user_name: str):
logger = get_run_logger()
<http://logger.info|logger.info>("Hello %s!", user_name)
@flow
def hello(user: str = "world"):
say_hi(user)
Deployment(
flow=hello,
name="s3_file_packager_with_remote_s3fs",
packager=FilePackager(
filesystem=RemoteFileSystem(basepath="<s3://prefect-orion/flows/>")
),
)
if you can't set up your endpoint properly with settings on RemoteFileSystem, please LMK and I'll open an issue
d
davzucky
07/22/2022, 11:08 AM
Good point. Forgot about that one using fsspec. Let me try it
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.