https://prefect.io logo
#prefect-community
Title
# prefect-community
e

Emma Rizzi

06/06/2022, 1:18 PM
Hi! i'm trying to migrate from a docker agent to a kubernetes agent, with Prefect 1.0. I have multiple VMs configured with NFS to access and store data I used a dockerstorage, and the
--volume
option on the docker agent for my flow to access shared volumes. I tried to configure this in the job_template option of the Kubernetes run (code in thread) but it fails with the following error when i launched the flow. Is there any example to follow to use volumes with kubernetes ?
Copy code
flow.run_config = KubernetesRun(
    job_template={
        "apiVersion": "batch/v1",
        "kind": "Job",
        "spec": {
            "template": {
                "spec": {
                    "containers":  {
                        "volumeMounts": [
                            { "name": "nfs", "mountPath": "/mnt/dev" },
                            { "name": "eodata", "mountPath": "/eodata" }
                        ],    
                    }, 
                    "volumes": [
                        {
                            "name": "nfs",
                            "hostPath": {
                                "path": "/mnt/dev",
                                "type": "Directory"
                            }
                        },
                        {
                            "name": "eodata",
                            "hostPath": {
                                "path": "/eodata",
                                "type": "Directory"
                            }
                        }
                    ]
                }
            }
        }
    }
)
k

Kevin Kho

06/06/2022, 2:48 PM
Maybe you can run this with debug log level and
--show-flow-logs
on the agent so we can get a better error message?
e

Emma Rizzi

06/06/2022, 3:03 PM
Thanks! I saw the error was due to my template formatting, containers expected a list and i used a dictionnary. It works as intended with the following template :
Copy code
job_template={
        "apiVersion": "batch/v1",
        "kind": "Job",
        "spec": {
            "template": {
                "spec": {
                    "containers":  [
                        {
                            "name": "flow",
                            "volumeMounts": [
                                { "name": "nfs", "mountPath": "/mnt/dev" },
                                { "name": "eodata", "mountPath": "/eodata" }
                            ]
                        }
                    ],    
                    "volumes": [
                        {
                            "name": "nfs",
                            "hostPath": {
                                "path": "/mnt/dev",
                                "type": "Directory"
                            }
                        },
                        {
                            "name": "eodata",
                            "hostPath": {
                                "path": "/eodata",
                                "type": "Directory"
                            }
                        }
                    ]
                }
            }
        }
    }
k

Kevin Kho

06/06/2022, 4:07 PM
Ah I see. Thanks for circling back!
👌 1
6 Views