Why are there only `get/put_directory` methods for filesystem, but no `get/put_file` methods? `fss...

Paweł Biernat

05/30/2024, 1:25 PM

Why are there only

get/put_directory

methods for filesystem, but no

get/put_file

methods?

fsspec

seems to support these. If I have a pipeline that requires a single file as an input do I have to download the entire folder?

Matthew Bell

05/30/2024, 2:41 PM

Speaking for GCS specifically but I imagine it's the same across the board, directories and files are considered the same thing.

Matthew Bell

05/30/2024, 2:42 PM

directories are just blobs with a specific naming convention that it uses to format how it displays data

Paweł Biernat

05/30/2024, 5:46 PM

Oh wow, I can just point it to a file instead of a directory and it works! Thanks!

Paweł Biernat

05/30/2024, 5:49 PM

I was thrown off by the name and didn't think to try it on a file.

Paweł Biernat

05/30/2024, 6:13 PM

Is there a pattern where I don't have to specify the file system type and just load it from the block type? For example, currently I have

Copy code

fs = SMB.load("azure-file-share")

while I'd like something like

Copy code

fs = RemoteFileShare.load("azure-file-share")

pass

"azure-file-share"

as a parameter to the flow to future proof the pipeline against swapping storage backend type. Does something like this make sense?

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.