Thread
#prefect-community
    m

    Marwan Sarieddine

    8 months ago
    Hi folks, question about the lazarus process. Why would lazarus try to reschedule a flow run if it reaches a successful state ?
    please see a screenshot of our flow run’s logs
    We are running on prefect cloud, with a kubernetes agent and dask executor on EKS
    Anna Geller

    Anna Geller

    8 months ago
    Could you DM me the flow run ID of this flow so that I can cross-check? In general, Lazarus is typically only rescheduling flow runs in case of some infrastructure issues (e.g. if your pod crashed). This is indeed a bit puzzling given that the flow run seems to have been successfully completed.
    m

    Marwan Sarieddine

    8 months ago
    (sent) thank you for taking the time to look into this
    Anna Geller

    Anna Geller

    8 months ago
    as a workaround, you could disable Lazarus for this flow as described here
    m

    Marwan Sarieddine

    8 months ago
    yes sure
    I should note that this is a rare occurrence, this flow run has been running successfully without being rescheduled by lazarus
    Anna Geller

    Anna Geller

    8 months ago
    Gotcha. Could it be that there are some unclosed DB connections or other resources like HTTP clients that you use in your flow? I saw a similar issue occurring due to resources failing to close/shut down. I’m also wondering if the concurrency limits you use may be related to this. I don’t see any clear indicators why this may be happening and would open a Github issue for now and suggest to disable Lazarus for this process as a workaround until then
    m

    Marwan Sarieddine

    8 months ago
    Could it be that there are some unclosed DB connections or other resources like HTTP clients that you use in your flow
    there are no DB connections, just a call to the prefect KV store to add a key and then remove it
    I’m also wondering if the concurrency limits you use may be related to this.
    there are no concurrency limits set