Christian Nuss

    Christian Nuss

    6 months ago
    hey all again! Do you have recommendations on how to handle an Agent killed due to OOM? We currently see Flows stay in a
    Running
    state, then we do a cancel via the API, which leaves them in Cancelling state
    Kevin Kho

    Kevin Kho

    6 months ago
    Hey @Christian Nuss, why type of agent is that?
    Christian Nuss

    Christian Nuss

    6 months ago
    its a bit jerry rigged at the moment, currently its a
    local
    agent
    jerry rigged == running a prefect agent in local mode on Kubernetes, so everything is happening in a single container 🤦‍♂️ (working on fixing that), so its no surprising we're OOM'ing, but i'd like to get flows in a good final state before proceeding
    Kevin Kho

    Kevin Kho

    6 months ago
    I think this will fix anyway when you move to the k8s agent. The Flow has a heartbeat that consistently pings Prefect to make sure it is alive. When you run into OOM, the flow and heartbeat will die, and then Prefect will mark the Flow as failed after enough missed heartbeats, so really Prefect should be able to mark it as Failed eventually and you should not need to edit the state. Does that make sense?
    Christian Nuss

    Christian Nuss

    6 months ago
    total sense, thank you! how many heartbeats does it take? something i cancelled 3 hrs ago is still
    Cancelling
    Kevin Kho

    Kevin Kho

    6 months ago
    Well this cancelling I am unsure it will resolve so I would just mark it as Failed. The heartbeats missing should mark it as failed in like 20 mins I think
    Christian Nuss

    Christian Nuss

    6 months ago
    curious: which piece of the stack is responsible for this? graphql, apollo or towel?
    Kevin Kho

    Kevin Kho

    6 months ago
    Towel
    See this
    Christian Nuss

    Christian Nuss

    6 months ago
    thank you! any tips on how to
    mark it as failed
    ?
    Kevin Kho

    Kevin Kho

    6 months ago
    There is a set state in the UI where you can set the state of a Task or Flow
    Christian Nuss

    Christian Nuss

    6 months ago
    ooo nice thanks!