Hi, I am submitting

spark-submit

jobs. The jobs are submitted on fire & forget mode (

spark.kubernetes.submission.waitAppCompletion=false

). I have a separate monitoring task to monitor status of each job. However as number of jobs increases, the cluster gets overwhelmed. Is there a way to throttle the task submits on the mapped items?

Can you share the flow code you try to optimize this way? At first glance, this looks more like something you may better address with a Kubernetes cluster autoscaler than with Prefect. Prefect can submit your tasks to the cluster but has no knowledge of the actual cluster resource utilization, so it would be hard to throttle it this way from Prefect

If you are on Cloud, then task run concurrency can help you. Orion also has task run concurrency limiting already. If your task can wait for the job to complete, this should be enough but if it can’t, then you may need to address on the Kubernetes side like Anna is suggesting

Thanks. I think for now K8s autoscaler might be way to go.