I am attempting to create a custom storage block Digital Oce

Question

I am attempting to create a custom storage block Digital Ocean uses an `https ` prefix but `s3 ` is hardcoded in the S3 filesystem object I did this by copying the S3 storage object code and removing the hardcoded prefix but leaving everything else the same except name and credential arguments The block successfully registers to prefect cloud However when I try to attach it as the storage block I get the error in the comments From what I can tell this is happening during the validation step At some point prefect is looking at the registry for the `Block` type via ```from prefect utilities dispatch import lookup type get registry for type from prefect blocks core import Block registry = get registry for type Block print registry ``` which returns all the prefect defined blocks If I try to build a deployment it will fail because the custom filesystem class I created does not show up in that registry Currently the remote file system object stores all settings as plain text which isn t ideal because I m storing access credentials Thoughts on the best way to proceed

Rio McMahon · Answer

stack trace ```Traceback most recent call last File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect cli utilities py line 41 in wrapper return fn args kwargs File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect utilities asyncutils py line 193 in wrapper return run async in new loop async fn args kwargs File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect utilities asyncutils py line 140 in run async in new loop return anyio run partial fn args kwargs File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages anyio core eventloop py line 70 in run return asynclib run func args backend options File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages anyio backends asyncio py line 292 in run return native run wrapper debug=debug File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 asyncio runners py line 43 in run return loop run until complete main File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 asyncio base events py line 587 in run until complete return future result File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages anyio backends asyncio py line 287 in wrapper return await func args File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect cli deployment py line 449 in build template = await Block load storage block File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect blocks core py line 514 in load return cls from block document block document File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect blocks core py line 401 in from block document else cls get block class from schema block document block schema File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect blocks core py line 432 in get block class from schema return lookup type cls block schema to key schema File Users riomcmahon opt anaconda3 envs goodkiwi env lib python3 7 site packages prefect utilities dispatch py line 187 in lookup type f No class found for dispatch key dispatch key r in registry for type KeyError No class found for dispatch key dostorage in registry for type Block ```

Rio McMahon · Answer

Solution ended up needing to define a prefect collection and installing into my python environment

Anna Geller · Answer

nice work figuring that out one user has recently contributed this <https github com PrefectHQ prefect pull 6344> in case you would also like to contribute your Digital Ocean Block

Anna Geller · Answer

it s a pretty straightforward interface and given you already figured it out might be useful to submit a PR to the main package up to you

Rio McMahon · Answer

< Anna Geller> I am struggling a little bit to actually implement it because DigitalOcean uses an `https ` prefix instead of `s3 ` although it is S3 compatible Based on I think you should be able to pass in the `endpoint url` client kwargs to fix this however in the scheme is automatically inferred from the URL provided and it returns an `fsspec implementations http HTTPFileSystem` implementation instead of `s3fs core S3FileSystem` implementation like I need Any thoughts on fixing this I m trying to manually edit the prefix to `s3 ` to trick the `RemoteFileSystem filesystem` property but this is pretty hacky

Rio McMahon · Answer

The main difference between the PR linked above and this is that `smb` is an accepted schema see here <https github com fsspec filesystem spec blob master fsspec registry py L179 L181> but since DO uses `https` it gets wonky

Rio McMahon · Answer

The relevant seems to be working code is ```class DOStorage ReadableFileSystem WritableFileSystem Block Store data as a file on Digital Ocean Spaces block type name = DOStorage logo url = <https cdn iconscout com icon free png 256 digital ocean 3521385 2944829 png> bucket path str = Field description= A DigitalOcean bucket path without `https ` prefix example= my bucket a directory within region name str = Field description= Digital ocean region name example= sfo3 access key SecretStr = Field None title= DO Access Key ID access key secret SecretStr = Field None title= DO Secret Access Key remote file system RemoteFileSystem = None property def basepath self gt str return f s3 self bucket path this will get fsspec to properly infer s3 object storage property def filesystem self gt RemoteFileSystem settings = unpack secrets if self access key settings key = self access key get secret value if self access key secret settings secret = self access key secret get secret value unpack endpoint url to solve compatibility issues settings client kwargs = endpoint url f https self bucket path region name self region name build the remote filesystem object without endpoint url self remote file system = RemoteFileSystem basepath=self basepath settings=settings return self remote file system``` wherein the `basepath` method I just manually prepend `s3 ` A less hacky solution I think would be to modify the `RemoteFileSystem` class to accept a kwarg that allows users to manually specifiy the fsspec scheme Thoughts on that

Anna Geller · Answer

in that case perhaps it makes sense to make a PR to fsspec first to fix that

Anna Geller · Answer

it s the first remote FS I see that doesn t follow this s3 gcs az convention

Anna Geller · Answer

I d prefer not to modify RemoteFileSystem block so contributing without making changes to that would be easier

Rio McMahon · Answer

Sounds good It is pretty annoying that DO doesn t conform to that naming convention but so it goes woman shrugging either way the above code snippet works as a modification for anyone running into this in the future

Rio McMahon · Answer

I m also not actually sure that this should be an fsspec change though `RemoteFileSystem` is the one shown in the following snippet that is the one that is incorrectly making the inference heavy finger quotations around incorrectly because ultimately it is DO s issue in using https not s3 as the scheme ```scheme = urllib parse urlsplit self basepath self filesystem = fsspec filesystem scheme self settings ``` If `fsspec filesystem ` was called with the `s3` scheme then it wouldn t be an issue I don t really think there is any way to make a PR to fsspec that would cover this behavior since they have users explicitly provide the scheme