Hi Prefect Experts, my company needs me to migrate Spark batch jobs to Prefect.io (schedule them via Prefect.io) , I have managed to set up Prefect-server on our Kubernetes cluster. Those Spark batch jobs are already configured to deploy via Helm to Kubernetes cluster , what is the easiest way to schedule them via Prefect.io on Kubernetes cluster ?
j
josh
02/24/2021, 3:18 PM
Hi @Ajith Kumara Beragala Acharige Lal if you are still intending to deploy the jobs via helm on k8s I would recommend looking in to how to deploy helm from Python (perhaps there are libraries for this) that you can call from inside a task or using something like the ShellTask to execute the helm commands from the shell
a
Ajith Kumara Beragala Acharige Lal
02/24/2021, 3:30 PM
Hi @josh, thanks , probably I should download the files to the pod?
Ajith Kumara Beragala Acharige Lal
02/24/2021, 3:30 PM
if its not via helm - then I should base on YAML ( job-template)?