Hello! Is there any way in Prefect to use a mount ...
# ask-communitya
Alejandro
10/21/2022, 3:10 PMHello! Is there any way in Prefect to use a mount point ("/home/user/mount/central-storage") as remote storage block? I have tried to set it to 'basepath' of LocalFileSystem, but this way I cannot run deployments on remote agents. Is there a "hacky" way to use RemoteFileSystem for this purpose?
✅ 1
z
Zanie
10/21/2022, 3:23 PM
I don’t think there’s a great way to get this done yet. Would you mind opening an issue so we can track this? I think it’s a very valid use-case.
Zanie
10/21/2022, 3:32 PM
You can hack it by wrapping it in a meta file system 😮
RemoteFileSystem(basepath="cached://~", settings={"target_protocol": "file"})a
Alejandro
10/21/2022, 3:47 PMOf course! Tomorrow at the latest I will open the issue. Thanks for the quick response!
Alejandro
10/21/2022, 3:49 PMWould you mind including a small snippet to see how the fix you propose works? I mean, where do I specify the mount path?
z
Zanie
10/21/2022, 3:52 PM
Instead of
~t
Talajasi
10/21/2022, 4:05 PMThat's what I had suspected, but it didn't work out for me (I'm writing from another account)
Talajasi
10/21/2022, 5:41 PMShould I specify in the deployment CLI command the block slug corresponding to this exact RemoteFileSystem? Because it's not working this way
z
Zanie
10/21/2022, 5:43 PM
I'm not sure, I didn't try it with a deployment. I just checked that it could be used to read a file. I won't be at a computer again for the rest of today.
t
Talajasi
10/21/2022, 6:06 PMOkey, me neither. I'll come back to this thread on Monday. Thanks!
a
Alejandro
10/24/2022, 8:54 AMHello again @Zanie! I opened the issue just a while ago. For the moment, is there any workaround to move forward in the meantime? Did you check if your solution worked when building a deployment?
Alejandro
10/28/2022, 2:05 PMHi @Zanie! Sorry for harping so much on this thread, but being able to work with mount as a shared storage point is an important feature for my project and I was wondering if there is any way to do it with Prefect (at least until this feature is officially included in the library). Thank you for your patience.
z
Zanie
10/28/2022, 4:39 PM
Hey! Did you try the workaround I provided?
Zanie
10/28/2022, 4:39 PM
I’m working on a lot of other things and haven’t been able to investigate this further yet.
a
Alejandro
10/31/2022, 8:49 AMThank you for your response @Zanie. I guess you are having a lot of work. I tried the alternative but it didn't work for me for a deployment. Anyway, shouldn't it work for a LocalFilesystem block? In essence, a mount point is accessed through a regular path.
z
Zanie
10/31/2022, 2:33 PM
Yeah it should work, but I think we might have some safety checks that will prevent it from being uesd.
Zanie
10/31/2022, 2:34 PM
If you want to open a PR that adds a bypass to those checks, that’d be okay with me. I know people use local storage with Docker and upload it themselves so I imagine the same thing would apply here?
a
Alejandro
11/08/2022, 9:10 AMExactly, I have an Orion server on an "X" machine and several Prefect Agents running on another "Y" machine. The machines have access to shared storage through a mount point. The agents run ETL pipelines, which are isolated deployments in Docker containers. My problem is that the agents are not able to reach the deployments code because I can't connect Orion to the agents via the shared path.