https://prefect.io logo
Title
d

Devin McCabe

06/22/2021, 3:59 PM
Anyone know how to kill a Cloud flow run that just won't die? I exhausted my free plan quota in the middle of a large mapped task and started getting thousands of "usage exceeded" messages. We'll obviously upgrade our plan, but that flow run won't seem to die. I've tried canceling it and manually setting its state but the "duration" continues to advance. (run ID is
42036d0b-6443-4648-8c93-50c2ac6a360c
)
n

nicholas

06/22/2021, 4:20 PM
Hi @Devin McCabe - if Prefect isn't able to cancel your run the easiest way to stop it would be to tear down the job/process that's running your flow, which will depend on your setup
d

Devin McCabe

06/22/2021, 4:25 PM
Yeah, it was ECS+Dask and everything is already stopped.
I can try stopping the agent, too
n

nicholas

06/22/2021, 4:27 PM
Sorry just to check, after tearing down the container your flow is still trying to set states? 🤔
If it's just the duration still climbing in the UI that's fine, the duration will cap when the run fails its heartbeat check or enters a finished state; it's a calculated duration that is based on the run
start_time
and
end_time
- if the run is still in a
Cancelling
state, it hasn't finished from what Cloud can tell, even if nothing is running. You can manually set this state to
Finished
or
Failed
to cap this
d

Devin McCabe

06/22/2021, 4:29 PM
No, it's effectively not doing anything anymore (log has stopped, all tasks failed or canceled). The only indication it's not fully stopped is that the duration isn't stopping like seen here:
Ok, got it. Won't worry about it then
n

nicholas

06/22/2021, 4:30 PM
Ok let me know if you're still having trouble with it and we can try to triage further.