hey all again! Do you have recommendations on how to handle an Agent killed due to OOM? We currently see Flows stay in a

Running

state, then we do a cancel via the API, which leaves them in Cancelling state

Hey @Christian Nuss, why type of agent is that?

its a bit jerry rigged at the moment, currently its a

local

agent

I think this will fix anyway when you move to the k8s agent. The Flow has a heartbeat that consistently pings Prefect to make sure it is alive. When you run into OOM, the flow and heartbeat will die, and then Prefect will mark the Flow as failed after enough missed heartbeats, so really Prefect should be able to mark it as Failed eventually and you should not need to edit the state. Does that make sense?

total sense, thank you! how many heartbeats does it take? something i cancelled 3 hrs ago is still

Cancelling

Well this cancelling I am unsure it will resolve so I would just mark it as Failed. The heartbeats missing should mark it as failed in like 20 mins I think

curious: which piece of the stack is responsible for this? graphql, apollo or towel?

Towel

thank you! any tips on how to

mark it as failed

?

There is a set state in the UI where you can set the state of a Task or Flow

ooo nice thanks!