Thread
#prefect-server
    s

    Sharath Chandra

    5 months ago
    Hi, I am submitting
    spark-submit
    jobs. The jobs are submitted on fire & forget mode (
    spark.kubernetes.submission.waitAppCompletion=false
    ). I have a separate monitoring task to monitor status of each job. However as number of jobs increases, the cluster gets overwhelmed. Is there a way to throttle the task submits on the mapped items?
    Anna Geller

    Anna Geller

    5 months ago
    Can you share the flow code you try to optimize this way? At first glance, this looks more like something you may better address with a Kubernetes cluster autoscaler than with Prefect. Prefect can submit your tasks to the cluster but has no knowledge of the actual cluster resource utilization, so it would be hard to throttle it this way from Prefect
    Kevin Kho

    Kevin Kho

    5 months ago
    If you are on Cloud, then task run concurrency can help you. Orion also has task run concurrency limiting already. If your task can wait for the job to complete, this should be enough but if it can’t, then you may need to address on the Kubernetes side like Anna is suggesting
    s

    Sharath Chandra

    5 months ago
    Thanks. I think for now K8s autoscaler might be way to go.