https://prefect.io logo
Channels
Powered by
c

Christine Chen

08/02/2023, 11:45 PM
Hi, guys! I am currently deploying a flow with a k8s job block and a docker image. If my flow needs to read/write data to an NFS, how do i do that? I tried mounting a volume in the job manifest but that didn’t work.
✅ 1
prefect duck 1
h

Henning Holgersen

08/03/2023, 7:30 AM
I have had the same issue and would love to see a working example. In the meantime, because the disk I wanted to read from supported SMB, I could write to it using fsspec. Generally, fsspec can read/write to a lot of stuff.
👀 1
c

Christine Chen

08/03/2023, 8:17 PM
Actually, mounting volumes via the job manifest is working for me now. I’m guessing the possible causes to my previous crashed runs could be that my PVC was in a different namespace from the prefect server, and also the path used within the flow code was incorrect.
One followup question: is there a way to customize the base job manifest via the KubernetesJob API?
h

Henning Holgersen

08/04/2023, 6:58 AM
Awesome! Feel free to share some minimal code to show it. My unsuccessful attempt at mounting shows how to add a patch when registering a kubernetesJob with python. https://github.com/radbrt/orion_flows/blob/main/blocks/k8svolume_job.py
c

Christine Chen

08/04/2023, 3:19 PM
@Henning Holgersen Thanks! I edited the job manifest from the UI after creating it. I tried applying customization as you do in your code, but oddly the changes didn’t show up in the job manifest.
h

Henning Holgersen

08/04/2023, 3:30 PM
@Christine Chen hmmm... interesting. As you can tell, this isn't my home field, and a part of me is a little relieved it isn't just me who can't make it work. But I was able to add a label using patches, if you check the bottom of this post: https://discourse.prefect.io/t/use-aks-workload-identity/3354. Wonder what the difference might be. It might be some issue about the path of the patch, I struggled with that one in the label example.
r

redsquare

08/04/2023, 3:40 PM
@Henning Holgersen I think you need
Copy code
"volumeMounts": [{
    "mountPath": "/logs",
    "name": "logs-data"
 }]
in your patch
actually ignore me
c

Christopher Boyd

08/04/2023, 3:50 PM
I’d expect you could do this just mounting a volume and a volumeMount?
Copy code
volumeMounts:
    - name: nfs-vol
      mountPath: /var/nfs # The mountpoint inside the container
  volumes:
  - name: nfs-vol
    nfs:
      server: 192.168.200.90 # IP to our NFS server
      path: /nfs # The exported directory
You can do sidecar containers which is basically the same
from the perspetive of the container, it’s just a networked filesystem to a path
If you can do this without prefect in general (just as a test case with your cluster), then you can convert the job spec to json and add it to the work pool
c

Christine Chen

08/04/2023, 7:37 PM
It worked for me by adding the manifest json to the job argument in KubernetesJob
🚀 1
h

Henning Holgersen

08/04/2023, 8:07 PM
@Christine Chen great to hear. If you have a code snippet you can share, I’d be interested in seeing it.
c

Christine Chen

08/04/2023, 8:10 PM
Yeah sure it’s like:
Copy code
import yaml
from prefect.infrastructure import KubernetesJob

with open("k8s-job.yaml", "r") as job_config:
    job_manifest = yaml.safe_load(job_config)

k8s_job = KubernetesJob(
    image=<image>,
    image_pull_policy="Always",
    pod_watch_timeout_seconds=600,
    finished_job_ttl=300,
    job=job_manifest,
)

k8s_job.save("k8s-flow", overwrite=True)
k8s-job.yaml
Copy code
apiVersion: batch/v1
kind: Job
metadata:
  labels: {}
spec:
  template:
    spec:
      parallelism: 1
      completions: 1
      restartPolicy: Never
      containers:
      - name: prefect-job
        env: []
        volumeMounts:
        - mountPath: <mount-path>
          name: <volume-name>
      volumes:
      - name: <volume-name>
        persistentVolumeClaim:
          claimName: <claim-name>
🙏 1
2 Views
Powered by