I am testing Prefect deployed into a kind kubernetes cluster using the helm chart. I'd like to mount a folder on my local filesystem to a running job. I've configured this in the job template using:

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job
  labels: {}
spec:
  template:
    metadata:
      labels: {}
    spec:
      containers:
        - name: flow
          image: prefecthq/prefect:latest
          imagePullPolicy: IfNotPresent
          ...
          volumeMounts:
            - name: filesystem-dir
              mountPath: /job-files
      volumes:
        - name: filesystem-dir
          hostPath:
            # directory location on host
            path: /path/to/my/dir
            # this field is optional
            type: Directory

I'm unable to mount the path which I believe is a function of the volume not being available on the agent. Is there a straightforward way for me to mount a volume to the agent in the helm chart deployment?

Hi @Jacqueline Riley Garrahan, I wonder whether mounting a volume is the right approach here. Even if we manage to do that successfully, what happens when you want to deploy your Server and your flows to a real distributed Kubernetes cluster? It wouldn't be possible to mount the files from your local file system then. What may be actually much easier and better for your PoC would be to copy your files to the docker image and specify the image on your Kubernetes job template. Here is how it could look like: #1 Create a Dockerfile

FROM prefecthq/prefect:1.0.0-python3.8
WORKDIR /opt/prefect
COPY your_dir/ /opt/prefect/your_dir/

#2 Build and push to some container registry:

docker build -t yourorg/your_image:latest .
docker push yourorg/your_image:latest

#3 Use this image in your job template:

apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job
  labels: {}
spec:
  template:
    metadata:
      labels: {}
    spec:
      containers:
        - name: flow
          image: yourorg/your_image:latest
          imagePullPolicy: IfNotPresent
          ...

Following this approach, there is no confusion where to mount a file directory (flow run pod vs agent pod) and your image can be even easier tested locally before deploying it to Kubernetes And btw, if you're just getting started with Prefect, Prefect Cloud is much easier to use than Prefect Server since you don't have to deploy anything other than your agent (which can be started with a single command), and it's free to use for 20,000 task runs every month, no credit card required. Check out this page for more info.

@Anna Geller in our case, we'll be deploying into an on-prem cluster and each resource will have access to the file system. Do you have any guidance on how to mount the volume using the helm chart?

if each machine on that cluster has access to that network file system, then mounting it on the run_config's job template would be the right place to do that rather than on the agent:

from prefect.run_configs import KubernetesRun

with Flow(
    name="your_flow", run_config=KubernetesRun(job_template=your_job_template)
) as flow:

But what would be probably easier in such on-prem setup would be to use a Docker agent? You could then mount the volume directly on the agent and this would make it available to all flow run containers (.aws is just an example):

prefect agent docker start --label AGENT_LABEL --volume ~/.aws:/root/.aws

Thanks for your help and time @Anna Geller. In my case the issue was that I'm deploying into a kind cluster on my laptop, which requires the mount as well and was causing a

directory not found

error on the job deployment

So is it solved now? Did the same job template work for you without the kind cluster?

I had to modify the template by dropping the command as mentioned here: https://github.com/PrefectHQ/prefect/issues/4525