https://prefect.io logo
Title
a

Alejandro

10/21/2022, 3:10 PM
Hello! Is there any way in Prefect to use a mount point ("/home/user/mount/central-storage") as remote storage block? I have tried to set it to 'basepath' of LocalFileSystem, but this way I cannot run deployments on remote agents. Is there a "hacky" way to use RemoteFileSystem for this purpose?
1
z

Zanie

10/21/2022, 3:23 PM
I don’t think there’s a great way to get this done yet. Would you mind opening an issue so we can track this? I think it’s a very valid use-case.
You can hack it by wrapping it in a meta file system 😮
RemoteFileSystem(basepath="cached://~", settings={"target_protocol": "file"})
— I’d love to get a real fix in though
a

Alejandro

10/21/2022, 3:47 PM
Of course! Tomorrow at the latest I will open the issue. Thanks for the quick response!
Would you mind including a small snippet to see how the fix you propose works? I mean, where do I specify the mount path?
z

Zanie

10/21/2022, 3:52 PM
Instead of
~
t

Talajasi

10/21/2022, 4:05 PM
That's what I had suspected, but it didn't work out for me (I'm writing from another account)
Should I specify in the deployment CLI command the block slug corresponding to this exact RemoteFileSystem? Because it's not working this way
z

Zanie

10/21/2022, 5:43 PM
I'm not sure, I didn't try it with a deployment. I just checked that it could be used to read a file. I won't be at a computer again for the rest of today.
t

Talajasi

10/21/2022, 6:06 PM
Okey, me neither. I'll come back to this thread on Monday. Thanks!
a

Alejandro

10/24/2022, 8:54 AM
Hello again @Zanie! I opened the issue just a while ago. For the moment, is there any workaround to move forward in the meantime? Did you check if your solution worked when building a deployment?
Hi @Zanie! Sorry for harping so much on this thread, but being able to work with mount as a shared storage point is an important feature for my project and I was wondering if there is any way to do it with Prefect (at least until this feature is officially included in the library). Thank you for your patience.
z

Zanie

10/28/2022, 4:39 PM
Hey! Did you try the workaround I provided?
I’m working on a lot of other things and haven’t been able to investigate this further yet.
a

Alejandro

10/31/2022, 8:49 AM
Thank you for your response @Zanie. I guess you are having a lot of work. I tried the alternative but it didn't work for me for a deployment. Anyway, shouldn't it work for a LocalFilesystem block? In essence, a mount point is accessed through a regular path.
z

Zanie

10/31/2022, 2:33 PM
Yeah it should work, but I think we might have some safety checks that will prevent it from being uesd.
If you want to open a PR that adds a bypass to those checks, that’d be okay with me. I know people use local storage with Docker and upload it themselves so I imagine the same thing would apply here?
a

Alejandro

11/08/2022, 9:10 AM
Exactly, I have an Orion server on an "X" machine and several Prefect Agents running on another "Y" machine. The machines have access to shared storage through a mount point. The agents run ETL pipelines, which are isolated deployments in Docker containers. My problem is that the agents are not able to reach the deployments code because I can't connect Orion to the agents via the shared path.