Hey there, I had 2 flows that were running for a f...
# ask-communitym
Mitchell Bregman
03/09/2021, 4:11 PMHey there, I had 2 flows that were running for a few days without realizing they were using up all the concurrency slots I had available in Cloud. I am trying to cancel all the “late” runs and essentially reset my scheduler… I have tried to do so via the interface, but it does not seem to clear the late runs out.. further, as I try to cancel the 2 flows that have been running, I am stuck in a “Cancelling” state. Is there any way someone can help me reset the current state of my Cloud? I can certainly kill the k8 pod that is running, but would rather it happen via UI. Thanks!
j
Jenny
03/09/2021, 4:27 PMHi Mitchell - For the flows in a cancelling state, we make a best attempt to cancel flows but wait until running tasks are finished. Best practice would be to wait until they finish but if you think there's a chance they'll never finish, I believe other users have had success changing the state to cancelled or failed using the "Set State" button on the flow run page of the UI.
Jenny
03/09/2021, 4:28 PMTo cancel the late runs, have you tried toggling the schedule off (and then you can toggle back on again if needed)?
Jenny
03/09/2021, 4:29 PMIf that doesn't work let me know and I'll see what else we can do to fix it!
Jenny
03/09/2021, 4:32 PMAlso - we're switching to a success based pricing model soon which should mean you no longer have this issue!
m
Mitchell Bregman
03/09/2021, 6:07 PM@Jenny yes! I have turned the schedules for all flows off, however there are still “late runs” scheduled… ive tried “clear”-ing them, but not happening
Mitchell Bregman
03/09/2021, 6:08 PMthe 2 flows that have been running for a while are stuck in “cancelling” still
j
Jenny
03/09/2021, 6:21 PMOk @Mitchell Bregman - let me see if we can look into this one further!
m
Mitchell Bregman
03/09/2021, 6:26 PMthanks!
j
Jenny
03/09/2021, 6:56 PMHi @Mitchell Bregman - can you give me an idea of how long those runs have been stuck in a cancelling state? And did you try setting the state to cancelled or failed through the UI? And double-checking, your flow concurrency limit is 2?
m
Mitchell Bregman
03/09/2021, 7:03 PMits been cancelling for about 4 hrs
Mitchell Bregman
03/09/2021, 7:03 PMand yes, we have a concurrency limit of 2
j
Jenny
03/09/2021, 7:21 PMSorry - just want to triple check everything here - did you try setting the state to cancelled using the set state button - should have a dialog come up like this?
m
Mitchell Bregman
03/09/2021, 7:32 PMwell, i ended up killing the k8 pods of the long standing flows - it seemed to clear… what i am trying to do is remove all the “late runs”
Mitchell Bregman
03/09/2021, 7:33 PMi have turned off schedule for all flows, but they are still showing up as late runs
j
Jenny
03/09/2021, 7:37 PMOk. How many late runs do you currently have? And are those flows auto-scheduled or are any run manually (e.g. from the run page of the UI)? I think we may have found a bug with that clear button in the UI but would expect that toggling the schedule would work so want to try figure out what's happening there.
m
Mitchell Bregman
03/09/2021, 7:41 PMthey are all auto-scheduled, currently have 465 late runs
Mitchell Bregman
03/09/2021, 7:41 PMi have toggled off the schedule button on all these flow runs, and this solution has worked in the past… but not currently
j
Jenny
03/09/2021, 7:48 PMThanks. Could you possibly DM me a flow and flow run id so we can look into it further?
m
Mitchell Bregman
03/09/2021, 7:53 PMsent!
j
Jenny
03/09/2021, 7:57 PMThank you!