Thread
#prefect-server
    Clemens

    Clemens

    1 year ago
    Hi! We are running prefect-server on a Kubernetes cluster, I have the issue that sometimes the resources in our cluster are limited and a flow cannot spawn a pod due to insufficient resources. However what seems to happen is that prefect keeps on spawning new jobs essentially blocking each other. Is there a way to prevent a flow from starting if there is another flow having resource issues? Alternatively, because the issue is with only one flow here, is there a way to prevent a flowrun from starting when there is already another flowrun of the same flow running (or pending)?
    nicholas

    nicholas

    1 year ago
    Hi @Clemens - I don't think this is possible with Prefect Server (at least not natively); Prefect Cloud offers flow concurrency limiting out of the box which would let you limit the number of flows with any label (or combination of labels) to a finite number.
    (note that flow concurrency in Server is an issue that's been explored at length in the past)
    Clemens

    Clemens

    1 year ago
    thanks, i will have a look!
    alternatively, can I tell a flowrun not to bother to retry if it has problems spawning a pod or dask cluster?
    nicholas

    nicholas

    1 year ago
    Hm, this might be possible with a resource manager but I'm not entirely sure. Let me pose the question to the team and see if they have any ideas (it'll likely be a little bit since it's quite late or early for most of our team)
    Clemens

    Clemens

    1 year ago
    awesome, thanks!
    nicholas

    nicholas

    1 year ago
    Hi @Clemens - after chatting with the team I don't think this is something you'll be able to accomplish through Server; you could do something very similar with Automations in Prefect Cloud, removing or cancelling a run that does not start for some period of time (preventing retries)