Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hello! I have a very basic case: I want to list all folders in a minio (s3) bucket. Then make parallel tasks that count the number of files in each folder.

Can I achieve that via blocks? I see that the S3 block only has `get_folder` which will download the whole storage to my local drive. I don’t want that.

I see two alternative ways:
• Make an S3 block, use that.
• Make a secret block, in my code set up a minio connection, use that.
Which one is better?

The built-in S3 block is meant to be used for remote code storage, so it's probably not the best fit for this.

The <https://github.com/PrefectHQ/prefect-aws|prefect-aws> collection has an <https://prefecthq.github.io/prefect-aws/s3/#prefect_aws.s3.S3Bucket|S3Bucket block> with a few more capabilities - the <https://prefecthq.github.io/prefect-aws/s3/#prefect_aws.s3.s3_list_objects|s3_list_objects> method might do what you want. The block can use MinIO credentials, too.

`s3_list_objects` for some reason started listing everything recursively :disappointed:

Screenshot 2022-11-02 at 17.04.10.png

I came up with this. It works, but at this point I could be establishing my own connection, only using blocks to store secrets.

Am I doing it right?

Seems like listing folders in a bucket should be a very common and straightforward thing to do

Screenshot 2022-11-02 at 17.27.07.png

Seems like just using the minio client is easier