Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Alejandro
10/21/2022, 3:10 PMHello! Is there any way in Prefect to use a mount point ("/home/user/mount/central-storage") as remote storage block? I have tried to set it to 'basepath' of LocalFileSystem, but this way I cannot run deployments on remote agents. Is there a "hacky" way to use RemoteFileSystem for this purpose?
✅ 1
z
Zanie
10/21/2022, 3:23 PMI don’t think there’s a great way to get this done yet. Would you mind opening an issue so we can track this? I think it’s a very valid use-case.
You can hack it by wrapping it in a meta file system 😮
RemoteFileSystem(basepath="cached://~", settings={"target_protocol": "file"})a
Alejandro
10/21/2022, 3:47 PMOf course! Tomorrow at the latest I will open the issue. Thanks for the quick response!
Would you mind including a small snippet to see how the fix you propose works? I mean, where do I specify the mount path?
z
Zanie
10/21/2022, 3:52 PMInstead of
~t
Talajasi
10/21/2022, 4:05 PMThat's what I had suspected, but it didn't work out for me (I'm writing from another account)
Should I specify in the deployment CLI command the block slug corresponding to this exact RemoteFileSystem? Because it's not working this way
z
Zanie
10/21/2022, 5:43 PMI'm not sure, I didn't try it with a deployment. I just checked that it could be used to read a file. I won't be at a computer again for the rest of today.
t
Talajasi
10/21/2022, 6:06 PMOkey, me neither. I'll come back to this thread on Monday. Thanks!
a
Alejandro
10/24/2022, 8:54 AMHello again @Zanie! I opened the issue just a while ago. For the moment, is there any workaround to move forward in the meantime? Did you check if your solution worked when building a deployment?
Hi @Zanie! Sorry for harping so much on this thread, but being able to work with mount as a shared storage point is an important feature for my project and I was wondering if there is any way to do it with Prefect (at least until this feature is officially included in the library). Thank you for your patience.
z
Zanie
10/28/2022, 4:39 PMHey! Did you try the workaround I provided?
I’m working on a lot of other things and haven’t been able to investigate this further yet.
a
Alejandro
10/31/2022, 8:49 AMThank you for your response @Zanie. I guess you are having a lot of work. I tried the alternative but it didn't work for me for a deployment. Anyway, shouldn't it work for a LocalFilesystem block? In essence, a mount point is accessed through a regular path.
z
Zanie
10/31/2022, 2:33 PMYeah it should work, but I think we might have some safety checks that will prevent it from being uesd.
If you want to open a PR that adds a bypass to those checks, that’d be okay with me. I know people use local storage with Docker and upload it themselves so I imagine the same thing would apply here?
a
Alejandro
11/08/2022, 9:10 AMExactly, I have an Orion server on an "X" machine and several Prefect Agents running on another "Y" machine. The machines have access to shared storage through a mount point. The agents run ETL pipelines, which are isolated deployments in Docker containers. My problem is that the agents are not able to reach the deployments code because I can't connect Orion to the agents via the shared path.