Thread
#prefect-server
    m

    Maria

    3 months ago
    Hi, where can I configure Lazarus behaviour? We are getting this error
    A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
    while we are waiting for nodes to free up
    Kevin Kho

    Kevin Kho

    3 months ago
    What are you trying to do? Increase retries or turn Lazarus off or increase the delay?
    m

    Maria

    3 months ago
    increase retries and the delay
    Kevin Kho

    Kevin Kho

    3 months ago
    You can for Server, but not Cloud I think. Are you on Server?
    m

    Maria

    3 months ago
    yes, i’m on server
    Kevin Kho

    Kevin Kho

    3 months ago
    I think delay is hardcoded but the limit is here
    This is the hardcode
    is this where we can override?
    can we override it in executor? like some values with “PREFECT_SERVER__” prefix
    Kevin Kho

    Kevin Kho

    3 months ago
    Yeah I think
    PREFECT__SERVER__LAZARUS__RESURRECTION_ATTEMPT_LIMIT
    before you start server
    m

    Maria

    3 months ago
    what about for delay?
    Kevin Kho

    Kevin Kho

    3 months ago
    You have to change the hardcode before you re-deploy 😐
    m

    Maria

    3 months ago
    hm i see. Thanks!
    i think based on the values.yaml file, we might be able to override it with these env variables? “PREFECT__SERVER__LAZARUS__RESURRECTION_ATTEMPT_LIMIT” and “PREFECT__SERVER___SCHEDULER___SCHEDULER__LOOP___SECONDS”_
    Kevin Kho

    Kevin Kho

    3 months ago
    that’s not lazarus. that is the rate at which new flow runs are scheduled
    m

    Maria

    3 months ago
    ah ok. Got it. Thanks!
    Unfortunately configuring in the config.toml didn’t update the lazarus resurrection attempt limit
    it’s still showing default 3
    A Lazarus process attempted to reschedule this run 3 times without success. Marking as failed.
    this was what i added in the prefect config
    [services]
        [services.lazarus]
        resurrection_attempt_limit = 20
    Kevin Kho

    Kevin Kho

    3 months ago
    I think config is used for
    prefect server start
    but not for helm so you need to edit in the helm side of things?
    m

    Maria

    3 months ago
    hm i tried both but i’m still getting the same error message and didn’t increase the attempts