Hi I have search the docs for an answer but could not find m

Question

Hi I have search the docs for an answer but could not find much so I thought I would ask here How does the Prefect engine deal with submitted `KubernetesRun` based flows that remain `pending` for some reasons For example what happens if I try to submit a flow but there isn t enough resources available on my cluster at that moment From my experience I can see that those flows get re submitted and another pod is created after some time but what happens then Both pods will run if given the resources Is there a limit after which the engine kills the flow run because of being unable to run it properly

Anna Geller · Answer

< Sylvain Hazard> my understanding is that is responsible for such use cases It gracefully retries failures caused by factors outside of Prefect s control Kubernetes pods not spinning up due to resource constraints on a node is a great example of that Once every 10 min Lazarus searches for distressed flow runs and reschedules them you could see this in the flow run s logs Scheduled flow runs without submitted or running task runs will be rescheduled up to 10 times the 11th time the flow run is marked as failed

Sylvain Hazard · Answer

Thanks That is a pretty clean process I like it

Sylvain Hazard · Answer

Weirdly enough it looks like my Lazarus kills flow runs after only 3 retry attempts is it something that s configurable

Sylvain Hazard · Answer

I got something like this

Anna Geller · Answer

I see let me check that

Sylvain Hazard · Answer

Thanks

Anna Geller · Answer

yup looks like for Server 3 is actually the default this is from config toml ``` services lazarus resurrection attempt limit = 3```

Anna Geller · Answer

the whole section in config toml is called services ``` services host = 0 0 0 0 services apollo host = $ services host port = 4200 services graphql host = $ services host port = 4201 debug = false path = graphql disable access logs = false timeout keep alive = 5 services lazarus resurrection attempt limit = 3 services towel max scheduled runs per flow = 10```

Sylvain Hazard · Answer

Nice thanks a lot