:wave: I'm trying to use RemoteFileSystem block wi...
# prefect-community
šŸ‘‹ I'm trying to use RemoteFileSystem block with WebHDFS. I'm running into 2 problems: 1. When I call
RemoteFileSystem(basepath="<webhdfs://basepath>", settings=params).write_path(path="subpath/filename", content=content)
I get an exception like this:
Copy code
null for uri: <https://hdfs.example.com>:port/webhdfs/v1webhdfs%3A//basepath/subpath?op=MKDIRS
Which is unsurpring, since URL above is clearly not valid. The problem here is that `fsspec.WebHDFS`'s methods makedirs/mkdir expect a slash-prefixed path w/o scheme (in this case
), while
just passes
and expects fsspec to do the rest. 2. A lot of fsspec.AbstractFileSystem methods rely on fsspec.infer_storage_options, most importantly
, which is used by
calls urlsplit and returns
, i.e. strips away scheme+netloc. Here the problem is that netloc is whatever comes after the scheme and before the first slash, e.g. for
would get stripped away erroneosly. This means one would have to prepend an extra segment to basepath to be sacrificed to fsspec's implementation (e.g.
). I can subclass
and override
to solve both problems. But then I would, it seems, need to make this class definition available and import it before calling
. What I don't understand is how I can make an agent use this to fetch code. Right now it just fails with
KeyError: "No class found for dispatch key 'subclassname' in registry for type 'Block'."
. Any ideas? Maybe there is a better way of doing this that doesn't involve subclassing?
āœ… 1
Hi! Iā€™d welcome an issue / pull request to address the parsing issues with the remote file system. It seems a bit tricky though. You need the block to be imported to be usable. One way is to register your module as a plugin e.g. https://github.com/PrefectHQ/prefect-aws/blob/main/setup.py#L30-L31
Alright, I'll try it out and get back to you, thanks! Will create an issue when I'm done, hopefully will have a better understanding.
Adding an entrypoint worked. Created an issue: https://github.com/PrefectHQ/prefect/issues/6957