@Marvin is canceling a flow run from UI not supposed to clean the kubernetes jobs it started? I dont see that happening in prefect 2.14

It appears that cancelling a flow run from the UI in Prefect Version 2.14 does not always clean up Kubernetes jobs started by the flow. Here are some insights: 1. Documentation Insights: - The cancellation process may face issues if the infrastructure block for the flow run has been removed or altered, or if it does not support cancellation. - The Prefect team is working on improving the cancellation process, and you can enable enhanced cancellation by setting

PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION=True

. More detailed information can be found in the Prefect Flows Documentation. 2. GitHub Issues: - There are documented issues such as Flow cancellation does not delete Dask Scheduler & Worker pods which mention that cancelling a flow using the Cloud UI cancels the flow pod but does not send a shut down command to the Dask Scheduler/Worker pods, leaving them orphaned. - Another issue, Support For Shutting Down Compute When Flow is Cancelled, describes a broader problem where cancelled flows leave hardware or compute resources running and suggests that users may need to manually handle shut down. - The issue Support cancellation of active flow runs also discusses improving the cancellation process to handle ongoing tasks more effectively. To mitigate the issue, consider enabling the experimental enhanced cancellation or manually handling the termination of Kubernetes jobs as Prefect continues to refine this functionality.