Hi folks, I've written a prefect integration which helps to deploy and monitor spark jobs on kubernetes via spark-on-k8s-operator. You can just simply use your existing

SparkApplication

or

ScheduledSparkApplication

(will be reduced to SparkApplication at run time) manifests in these flows. Delegating the scheduling and retrial functionalities to prefect, it gives one much more flexibility to add other sorts of tasks sequentially or paralelly. Please let me know how it works! Thanks https://tardunge.github.io/prefect-spark-on-k8s-operator/ https://github.com/tardunge/prefect-spark-on-k8s-operator