Hello! I have a very basic case: I want to list al...
# data-tricks-and-tips
Hello! I have a very basic case: I want to list all folders in a minio (s3) bucket. Then make parallel tasks that count the number of files in each folder. Can I achieve that via blocks? I see that the S3 block only has
which will download the whole storage to my local drive. I don’t want that. I see two alternative ways: • Make an S3 block, use that. • Make a secret block, in my code set up a minio connection, use that. Which one is better?
The built-in S3 block is meant to be used for remote code storage, so it's probably not the best fit for this. The prefect-aws collection has an S3Bucket block with a few more capabilities - the s3_list_objects method might do what you want. The block can use MinIO credentials, too.
for some reason started listing everything recursively 😞
I came up with this. It works, but at this point I could be establishing my own connection, only using blocks to store secrets. Am I doing it right?
Seems like listing folders in a bucket should be a very common and straightforward thing to do
Seems like just using the minio client is easier