@Marvin do any of the Prefect.fileSystem blocks allow you to interact with a local folder, while running the workflow in a Docker runner? If yes, what steps are involved (ex: how to tell the runner to mount a volume)?

Yes, the Prefect

LocalFileSystem

block allows you to interact with a local folder. However, when running the workflow in a Docker runner, you have to ensure that the local directory is accessible to the Docker container. Here are the steps to do this: 1. Create a

LocalFileSystem

block with your desired base path.

python
   from prefect.filesystems import LocalFileSystem
   fs = LocalFileSystem(basepath="/foo/bar")

(Local File System Documentation) 2. While configuring your Docker container, you'll have to mount the volume that corresponds to the local folder you want to interact with. 3. Set the appropriate

basepath

in the

LocalFileSystem

block to the mount point inside the Docker container. For example, when running the Docker container, you can use the

-v

or

--volume

option to bind mount a volume:

sh
docker run -v /local/path:/container/path my-prefect-image

It's important to note that the execution environment for your workflows may not have the same file system as the environment you are writing and deploying your code on, hence access to the

LocalFileSystem

is limited to the exact path provided. This needs to be taken into account when mounting volumes in Docker. Be aware that the Prefect Docker documentation may provide additional information on networking and environment variables that might be necessary when using Docker runners: - Docker guide - Prefect Docker worker documentation Always refer to the latest documentation for the up-to-date steps and best practices.

For anyone who finds this after the fact, it looks like you can control this at the Worker level, as they have a "Volumes" configuration option (see attached screenshot). @Marvin,

Indeed, for those who come across this inquiry in the future, it is correct that the Prefect Worker has a "Volumes" configuration option that allows you to specify mounts when the worker is created. Therefore, if you require access to local filesystem directories, you can set up volume mounts in the worker's configuration. The "Volumes" option enables you to mount local paths inside your Docker containers, providing a way for your flows to interact with the local file system even when they are executed within Docker. When setting up your Prefect Worker, you would leverage the relevant configuration options to define the mounts. Make sure to map your local paths to the expected paths inside your container, analogous to the way you would specify volumes when running Docker with the

-v

or

--volume

option. For the most accurate and detailed instructions on how to properly set this up, consult the Prefect documentation on workers and the specific section regarding Docker workers and volumes: - Prefect Workers Documentation - Docker Volumes on Prefect Workers Remember that the documentation might evolve over time, thus it's advisable to always check for the latest updates which might not have been present at the time of this response.