https://prefect.io logo
Join Slack
Powered by
Hello all -- looking to see if anyone has a workin...
# ask-community
b
Hello all -- looking to see if anyone has a working Prefect 2.0 example of what it looks like to use a RemoteFileSystem block through the Cloud UI to setup SMB (including auth -- even if the auth can't be setup in the Cloud UI block itself) storage to use with flows deployment. Context: Our team doesn't have S3, GC, or Azure storage available to us. It all has to be kept on-premise. Our Data Center wants to provide us with SVM storage that's accessible via SMB (and it looks like fsspec does support a "smb://" connection string). Thanks!
✅ 1
a
SMB?
b
Sorry -- Samba :)
a
context?:)
what is it?
Sambda is a dance 💃
🎉 2
b
It's what we use to connect to Windows servers, or Windows based storage from Linux. Here is the fsspec api entry on it: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.smb.SMBFileSystem
a
btw if that's easier to manage, we plan to provide a more straightforward way to bake your flow code into your docker image
🙏 1
would you want to contribute a draft PR?
implementing it an fsspec-based file system block is quite straightforward based on RemoteFileSystem block
check S3, GCS, Azure . they follow the same interface https://github.com/PrefectHQ/prefect/blob/main/src/prefect/filesystems.py
you could contribute SMBFileSystem block
b
Making it easier to bake code into images will be good (since we do that some) -- but we try and use shared images as much as possible, since we have quite a few small flows, and appreciate how light weight just a script (and possibly a few small modules) can be.
👍 1
"you could contribute SMBFileSystem block" I am definitely willing to look into this. I've been interested in contributing something for a while.
Ok. Yeah, the structure of the storage blocks for the 3 cloud storage options more or less makes sense to me -- I briefly looked through the structure for the s3 block last week when I was looking into to what I would need to do to create a similar Azure block (for my own person projects, not work -- and prior to seeing your PR for it over the weekend that went into 2.0.2).
I'll look into creating a draft of something. First I'll need to tinker around with using fsspec for smb in general, then RemoteFileSystem -- my SMB connections up to this point have been using a different library (more focused on SMB than fsspec, but also not nearly as widely used, and likely not as dependable).
🙌 2
🙏 1
a
I believe most of work will be first testing out if you can use the already existing RemoteFileSystem block with SMB by specifying this as your base path:
Copy code
<smb://myuser:mypassword@myserver.com/your_base_path_dir/>
and then once you get that working, you can build a specific block for SMB by moving the sensitive data like username, password, host to "settings" variables on the settings dict it then should work it would be hard for me to contribute since I don't have access to such system to properly test and QA this, but if you contribute, feel free to tag or ping me for review 🙌
👍 1
and I can help answer any questions along the way, too
b
Sounds great. Appreciate you giving me a direction to head in.
🙌 2
I was able to get it working through just using the RemoteFileSystem block, and then was also able to get it working using the the same block, but with host, port, username, password in the settings dictionary. I created a scratch file and copied the storage block structure of S3, and it looks like it creates just fine in my cloud 2.0 instance. I can also pull 
<block>.load(...)
the block data from inside of the script. But, if I try to using 
deployment build -sb smb/<name>
I get the following: "ValueError: Unable to find block document named test-smb for block type SMB" Am I correct in assuming that to use my currently-custom block with 
-sb
it needs to be registered somewhere I don't have access to? Or is it something I should be able to reference, even if it's not an official storage block in Prefect 2.0 at the moment? I did register it with: https://docs.prefect.io/concepts/blocks/#registering-blocks EDIT: Obviously before I do any sort of pull request I'll follow the contributor guide, but just trying to thinker my way through it at the moment. EDIT 2: I also do see "SMB" in the "Type" drop down in the web UI (I've been digging through the prefect code based on stack traces looking for where that is pulled from). EDIT 3: One difference I can see between my storage block and others is that mine does not list any capabilities. While others, like S3, show "read-path" and "write-path".
a
your storage block should be read and write it's expected that the block cannot be immediately used in a deployment - it would need to be registered https://docs.prefect.io/concepts/blocks/#registering-blocks but if it works with RemoteStoreBlock, you can submit a PR and once it's merged, you should be able to use it right away in deployments
b
I created a draft PR for a SMB storage block: https://github.com/PrefectHQ/prefect/pull/6344#issue-1333882276
a
wow, that's awesome! I'll review
4 Views
Open in Slack
PreviousNext
Powered by