Tejal Singh
08/23/2022, 9:51 AMAnna Geller
08/23/2022, 10:57 AMspark_submit
command to submit jobs to the cluster and poll for statusTejal Singh
08/23/2022, 12:24 PM# Run on a Kubernetes cluster in cluster deploy mode
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master <k8s://xx.yy.zz.ww:443> \
--deploy-mode cluster \
--executor-memory 20G \
--num-executors 50 \
<http://path/to/examples.jar> \
1000
can we also connect to a K8 spark cluster like this using python code only and run execution:
conf = SparkConf().setAppName(appName).setMaster(<k8s://xx.yy.zz.ww:443>)
sc = SparkContext(conf=conf)
Anna Geller
08/23/2022, 1:38 PM