https://prefect.io logo
Title
m

Mia

06/09/2022, 4:40 PM
Hi, where can I configure Lazarus behaviour? We are getting this error
A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
while we are waiting for nodes to free up
k

Kevin Kho

06/09/2022, 5:06 PM
What are you trying to do? Increase retries or turn Lazarus off or increase the delay?
m

Mia

06/09/2022, 5:23 PM
increase retries and the delay
k

Kevin Kho

06/09/2022, 5:24 PM
You can for Server, but not Cloud I think. Are you on Server?
m

Mia

06/09/2022, 5:24 PM
yes, i’m on server
k

Kevin Kho

06/09/2022, 5:26 PM
I think delay is hardcoded but the limit is here
This is the hardcode
is this where we can override?
can we override it in executor? like some values with “PREFECT_SERVER__” prefix
k

Kevin Kho

06/09/2022, 5:29 PM
Yeah I think
PREFECT__SERVER__LAZARUS__RESURRECTION_ATTEMPT_LIMIT
before you start server
m

Mia

06/09/2022, 5:30 PM
what about for delay?
k

Kevin Kho

06/09/2022, 5:31 PM
You have to change the hardcode before you re-deploy 😐
m

Mia

06/09/2022, 5:32 PM
hm i see. Thanks!
i think based on the values.yaml file, we might be able to override it with these env variables? “PREFECT__SERVER__LAZARUS__RESURRECTION_ATTEMPT_LIMIT” and “PREFECT__SERVER___SCHEDULER___SCHEDULER__LOOP___SECONDS”_
k

Kevin Kho

06/09/2022, 5:38 PM
that’s not lazarus. that is the rate at which new flow runs are scheduled
m

Mia

06/09/2022, 5:39 PM
ah ok. Got it. Thanks!
Unfortunately configuring in the config.toml didn’t update the lazarus resurrection attempt limit
it’s still showing default 3
A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
this was what i added in the prefect config
[services]
    [services.lazarus]
    resurrection_attempt_limit = 20
k

Kevin Kho

06/09/2022, 7:59 PM
I think config is used for
prefect server start
but not for helm so you need to edit in the helm side of things?
m

Mia

06/10/2022, 7:54 AM
hm i tried both but i’m still getting the same error message and didn’t increase the attempts