Sharath Chandra
04/25/2022, 4:08 AMspark-submit
jobs. The jobs are submitted on fire & forget mode (spark.kubernetes.submission.waitAppCompletion=false
). I have a separate monitoring task to monitor status of each job.
However as number of jobs increases, the cluster gets overwhelmed.
Is there a way to throttle the task submits on the mapped items?Anna Geller
Kevin Kho
Sharath Chandra
04/26/2022, 1:47 AM