Hi! We are running prefect-server on a Kubernetes cluster, I have the issue that sometimes the resources in our cluster are limited and a flow cannot spawn a pod due to insufficient resources. However what seems to happen is that prefect keeps on spawning new jobs essentially blocking each other. Is there a way to prevent a flow from starting if there is another flow having resource issues? Alternatively, because the issue is with only one flow here, is there a way to prevent a flowrun from starting when there is already another flowrun of the same flow running (or pending)?

Hi @Clemens - I don't think this is possible with Prefect Server (at least not natively); Prefect Cloud offers flow concurrency limiting out of the box which would let you limit the number of flows with any label (or combination of labels) to a finite number.

thanks, i will have a look!

Hm, this might be possible with a resource manager but I'm not entirely sure. Let me pose the question to the team and see if they have any ideas (it'll likely be a little bit since it's quite late or early for most of our team)

awesome, thanks!

Hi @Clemens - after chatting with the team I don't think this is something you'll be able to accomplish through Server; you could do something very similar with Automations in Prefect Cloud, removing or cancelling a run that does not start for some period of time (preventing retries)