@Marvin is there a way to terminate every single flow run that has been long running? My agent went down over the weekend and a lot of the flow runs are still in a running state.

To terminate flow runs that are long-running and appear to be stuck, you have a couple of options available: 1. Using the Command Line Interface (CLI): You can cancel a specific flow run by using the following command and passing the ID of the flow run:

bash
   prefect flow-run cancel 'FLOW_RUN_ID'

Replace

'FLOW_RUN_ID'

with the actual ID of the specific flow run that you would like to terminate. 2. Using the Prefect UI: You can manually navigate to the flow run's detail page in the Prefect UI and click on the cancel button there. However, if you have to cancel several flow runs, manually identifying and cancelling each of them might be cumbersome. You could automate this process by: - Writing a script that uses Prefect's APIs to fetch the flow runs that have been running for an unusually long time (based on a threshold you define). - Then, using these API endpoints or the CLI within the script, you could automatically send the cancellation command for each relevant flow run. Please note that there is also a feature under development for enhanced cancellation which can be enabled by setting

PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION=True

on your worker or agents. This could be useful for ensuring cancellations are more robust. To accomplish the automated cancellation, you may need to create a script that interacts with Prefect's API or CLI, identifying long-running flow runs and sending a cancel command to each. If the specific steps for identifying and terminating long-running flows through automation are needed, I could assist with that as well, albeit with a sigh of expectation for the problems it may bring.

@gperrone I have written a script that does this before, essentially what I do is poll for flows using

get_flow_runs_by_tags

and then compare

expected_start_time

to the current time, and if it's past a certain threshold manually update the state of the flow to

CANCELLED

(or

PENDING

if I want to rerun it) using

set_flow_run_state

. Hope that helps?

Thank you this is helpful @Gabrielle Bennett