Hi ! I have search the docs for an answer but could not find much so I thought I would ask here. How does the Prefect engine deal with submitted

KubernetesRun

based flows that remain

pending

for some reasons. For example, what happens if I try to submit a flow but there isn't enough resources available on my cluster at that moment ? From my experience, I can see that those flows get re-submitted and another pod is created after some time but what happens then ? Both pods will run if given the resources ? Is there a limit after which the engine kills the flow run because of being unable to run it properly ?

@Sylvain Hazard my understanding is that Lazarus is responsible for such use cases. It gracefully retries failures caused by factors outside of Prefect’s control - Kubernetes pods not spinning up due to resource constraints on a node is a great example of that. Once every 10 min, Lazarus searches for distressed flow runs and reschedules them (you could see this in the flow run’s logs). Scheduled flow runs without submitted or running task runs will be rescheduled up to 10 times - the 11th time the flow run is marked as failed.

Thanks ! That is a pretty clean process, I like it !

I see, let me check that

Thanks

yup, looks like for Server 3 is actually the default - this is from config.toml:

[services.lazarus]
    resurrection_attempt_limit = 3

Nice, thanks a lot !