https://prefect.io logo
m

Matthew Blau

02/15/2021, 3:39 PM
Hello all, I am looking through the documentation and I am struggling to find information on passing volume information to Docker Storage. My flow outputs some files to disk and I would like to mount a volume so that we can see the files that were created on the filesystem, outside of the docker container. How would I accomplish this?
s

Samuel Hinton

02/15/2021, 3:43 PM
I cant speak to how to mount up a certain directory, but in your specific case I would look at potentially saving these files by using an S3 bucket, which you can authenticate regardless of where in a container you are 🙂
k

Kyle Moon-Wright

02/15/2021, 5:07 PM
Hey @Matthew Blau I believe volume mounting is intended to be encapsulated on the Agent to apply to any submitted flow runs rather than the Storage object of the flow itself, and can be configured to multiple volumes. Of course as mentioned, persisting these files to another db is also viable.
a

ale

02/15/2021, 5:08 PM
Hey @Matthew Blau 🙂 If you use Docker Agent, you can use
--volume
to mount a volume. However, if you need to persist files I suggest to use S3/Azure Blob Storage, as @Samuel Hinton said
upvote 1
m

Matthew Blau

02/15/2021, 5:13 PM
@Kyle Moon-Wright I had seen that option for volume mounting and that it wants /host/path:/container/path. That would mean something like .:/app, correct? Our existing solution with a docker compose file does this that for volume mounting.
Copy code
volumes:
      - .:/app
The files that are being written are files that are then used to populate our db and having them easily accessible to the file system for checking that the correct info is populated in them is our use case here
k

Kyle Moon-Wright

02/15/2021, 5:26 PM
I think that will do it, as an example for all submitted flow runs:
Copy code
prefect agent docker start --volume /source/file:/target/file
Hopefully someone can verify 😅
upvote 1
m

Matthew Blau

02/15/2021, 9:16 PM
@Kyle Moon-Wright I have been successful in writing the exact directory that I wanted, but the issue is just that: it wants an exact directory and I run into a module not found error if I have it output files to a different directory the directory that the flow is located in; so it would seem that it does not scale very well for multiple projects that all write to their own directories
🤔 1
3 Views