https://prefect.io logo
t

Tim Wright

03/24/2022, 12:49 PM
Does anyone know if there is a way to adjust the Lazarus process polling interval? We are running our flows in an AKS Cluster (Kubernetes) that is set to automatically scale out. I've noticed that during periods of high Azure traffic, it can take our Cluster some time for the scaled out nodes to actually show up. What I've observed is that a flow run will schedule and then trigger a scale-up event on the cluster. The next node will become available to schedule flows >10 minutes later. It appears that Lazarus is scheduling another flow run - and then when the scaled up node becomes available 2 versions of the flow (the original and the lazarus scheduled flow) are both running. Has anyone else run into this? Or does anyone have guidance on how we can handle it? Currently we've disabled the Lazarus process, ideally we could stretch the interval from 10 minutes.
k

Kevin Kho

03/24/2022, 1:35 PM
Are you on Prefect Cloud or Prefect Server?
t

Tim Wright

03/24/2022, 1:36 PM
Prefect Cloud
k

Kevin Kho

03/24/2022, 1:40 PM
This is not configurable because you’d need to configure the interval of the Lazarus service, which we run globally for everyone.
t

Tim Wright

03/24/2022, 1:42 PM
Have you heard of others running into this issue? Is there an alternative mechanism to resolve the behavior aside from disabling Lazarus altogether?
I think we might be able to upgrade our AKS configuration to try to guarantee scale out events are handled more quickly. Was trying to understand if we could modify something at the application level to avoid that
k

Kevin Kho

03/24/2022, 1:55 PM
Yes I normally see people do you second suggestion of being more generous with scaling. Haven’t seen it happen commonly, but was the behavior better for you before? I think there may have been a recent change to the service
t

Tim Wright

03/24/2022, 1:56 PM
Not something I've experienced recently since Lazarus is disabled for us. Was just remembering to circle back to see if there was a known app-level solution
k

Kevin Kho

03/24/2022, 2:39 PM
Yeah unfortunately not really. There’s really only the on/off toggle
t

Tim Wright

03/24/2022, 2:45 PM
Ok. Appreciate the response @Kevin Kho
8 Views