Hello @Marvin, I have some flow runs that are stuck in cancelling and when I try the following script, it doesn't actually cancel them in the database, it still shows up! What can I do to remedy this?

from prefect import get_client
from prefect.server.schemas.states import State, StateType
from prefect.server.schemas.filters import FlowRunFilter, FlowRunFilterState, FlowRunFilterStateName, FlowRunFilterStateType
import asyncio

async def get_cancelling_flows():
    async with get_client() as client:
        
        state_filter = FlowRunFilter(state=FlowRunFilterState(type=FlowRunFilterStateType(any_=['CANCELLED'])))
        # Query all flow runs
        flow_runs = await client.read_flow_runs(flow_run_filter=state_filter)
        cancelled_state = State(type=StateType.CANCELLED)

        # Loop through each flow run and change its state to "CANCELLED"
        for flow in flow_runs:
            try:
                await client.set_flow_run_state(
                    flow_run_id=flow.id,
                    state=cancelled_state,
                    force=True
                )
                print(f"Flow run {flow.id} has been set to CANCELLED.")
            except Exception as e:
                print(e)

# Run the async function to get the cancelling flows
asyncio.run(get_cancelling_flows())

To help you with this issue, I need to first confirm which version of Prefect you are using, as the code syntax and API behavior can vary significantly between Prefect 1.x, 2.x, and 3.x. From the code snippet you provided, it looks like you're using either Prefect 2.x or 3.x, but I want to be sure. Could you please let me know the version of Prefect you are currently using? You can find this by running:

python
import prefect
print(prefect.__version__)

Once I have that information, I can provide the most accurate guidance and search for relevant documentation and issues that might help resolve the problem with flow runs getting stuck in a "cancelling" state.

@Marvin the prefect version is 2.19.3

When dealing with flow runs that are stuck in a "cancelling" state in Prefect 2.x, there are a few things to keep in mind: 1. Infrastructure Issues: As per the Prefect documentation, flow runs can get stuck in the cancelling state if there are issues with the infrastructure. For instance, if the infrastructure block associated with the flow run is altered, missing, or doesn't support cancellation, it can prevent proper cancellation. Additionally, if the infrastructure cannot be found or has already been terminated, the flow run might be marked as cancelled without actually completing the cancellation process. 2. Code Considerations: Ensure that the script you use is appropriately targeting the flow runs you wish to cancel. Your code attempts to set the state to "CANCELLED," but you should ascertain that the state types and settings in your script match the expectations of your Prefect setup. Here are a few steps you could consider for troubleshooting: - Check Infrastructure and Environment: Validate that the infrastructure associated with your flow runs is functioning as expected and supports cancellation. This may involve confirming container IDs, job names, or any infrastructure identifiers. - Script Improvement: The code you've shared is largely correct, but double-check the specific filters and state settings you're applying to ensure they align with your needs. Consider capturing more detailed exceptions to diagnose underlying errors. Refer to the Prefect documentation for more refined guidance on using the client to cancel flow runs effectively. You can find more detailed guidance in the Prefect 2.x documentation. If these steps do not resolve the issue, you can also file a bug or search for similar issues on the Prefect GitHub repository. If needed, I can assist with searching for related GitHub issues.