@Marvin why does cancelling a flow run not delete the underlying kubernetes pod

Cancelling a flow run in Prefect does not necessarily delete the underlying Kubernetes pod due to several reasons. Here are some insights based on Prefect's documentation and existing issues: 1. Infrastructure Block Support: - The flow run's infrastructure block might lack support for cancellation. If the associated infrastructure does not support cancellation, the Kubernetes pod will not be terminated automatically. 2. Mismatch in Identifier Scope: - Prefect uses an internal identifier (

infrastructure_pid

) to track the infrastructure associated with a flow run. A mismatch in this identifier or issues in tracking can result in unsuccessful cancellation attempts. 3. Orphaned Dask Pods: - If a flow run uses Dask for task execution, cancelling the flow via the Cloud UI without sending a shutdown command to the Dask Scheduler/Worker pods can leave these pods orphaned. The flow pod might be deleted immediately upon cancellation, leaving Dask pods running without performing any tasks. - Issue Example: Flow cancellation does not delete Dask Scheduler & Worker pods 4. Enhanced Cancellation: - Prefect is working on improving the cancellation process. By enabling the experimental feature

PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION

, Prefect might handle cancellations more effectively, including deleting associated Kubernetes pods.

sh
   prefect config set PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION=True

5. Monitoring and Deployment Requirements: - Cancellation requires the flow run to be associated with a deployment and monitored to enforce cancellation. Inline subflow runs, i.e., those created without

run_deployment

, cannot be cancelled without cancelling the parent flow run. For further details, you can refer to: - Prefect Docs on Cancellation - Relevant GitHub Issues: - Deleting a Flow with an Ongoing Flow Run on Kubernetes - DaskKubernetes Environment and AWS Auto-scaler In conclusion, ensuring your infrastructure supports cancellation and enabling experimental features can help address issues related to Kubernetes pods not being deleted upon flow run cancellation.