https://prefect.io logo
Title
c

Clemens

07/09/2021, 8:04 AM
Hi! We are running prefect-server on a Kubernetes cluster, I have the issue that sometimes the resources in our cluster are limited and a flow cannot spawn a pod due to insufficient resources. However what seems to happen is that prefect keeps on spawning new jobs essentially blocking each other. Is there a way to prevent a flow from starting if there is another flow having resource issues? Alternatively, because the issue is with only one flow here, is there a way to prevent a flowrun from starting when there is already another flowrun of the same flow running (or pending)?
n

nicholas

07/09/2021, 8:07 AM
Hi @Clemens - I don't think this is possible with Prefect Server (at least not natively); Prefect Cloud offers flow concurrency limiting out of the box which would let you limit the number of flows with any label (or combination of labels) to a finite number.
(note that flow concurrency in Server is an issue that's been explored at length in the past)
c

Clemens

07/09/2021, 8:09 AM
thanks, i will have a look!
alternatively, can I tell a flowrun not to bother to retry if it has problems spawning a pod or dask cluster?
n

nicholas

07/09/2021, 8:10 AM
Hm, this might be possible with a resource manager but I'm not entirely sure. Let me pose the question to the team and see if they have any ideas (it'll likely be a little bit since it's quite late or early for most of our team)
c

Clemens

07/09/2021, 8:11 AM
awesome, thanks!
n

nicholas

07/09/2021, 3:39 PM
Hi @Clemens - after chatting with the team I don't think this is something you'll be able to accomplish through Server; you could do something very similar with Automations in Prefect Cloud, removing or cancelling a run that does not start for some period of time (preventing retries)