I get the purpose of the lazarus process is to restart “distressed” flow runs in a constant interval of ~10 minutes. In our usecase we are running dask-kubernetes on AWS EKS and we are deploying the scheduler in “local” mode which it seems not to play well with lazarus (I have started testing this out today and I can’t get lazarus to run - even after waiting for 20 minutes on a distressed flow run).

So what happens is for a given flow run two main pod types/specs are used: a job pod where the scheduler is also run in and a worker pod where the dask workers are run. Our most common infrastructure-related issues occur due to cluster failing to schedule the worker pod not the job pod (due to a certain failure to provision resources (compute/volumes/secrets ..)). Unfortunately in this setup the lazarus process is never triggered given the flow run is hanging in a “Running” state instead of a “Submitted” state from what I can tell.

I am wondering if someone else has experienced this - or if I am missing something here

Let me check if the team has any ideas on this one.

will verify this in a bit

just one thing is in the old way of doing things with prefect Environments, the pod that has the CloudFlowRunner is the second pod to run and as such has a better experience with lazarus

so the docs on lazarus state:

"Where necessary, flow runs without submitted or running task runs will be rescheduled by the Lazarus process up to 10 times."
<https://docs.prefect.io/orchestration/concepts/services.html#how-does-it-work>

And sadly in our case of no running tasks - i.e. all tasks remain in pending mode we can’t really rely on lazarus …

So we do have access to flow-level SLAs via automations - it is far from ideal given with lazarus one would know something is up/ get notified after 10 minutes, versus in a long-running flow one won’t know there is an issue until the full expected run time has elapsed (i.e. up to 60~ 120 minutes in our case)