To ensure that a Kubernetes Job and the associated Pod are properly terminated when a Prefect run is canceled, Prefect uses a robust cancellation process. Here's a summary of how it works:
1.
Infrastructure Identification: Prefect identifies infrastructure types using unique identifiers:
- Kubernetes Jobs are identified by the Kubernetes cluster name and the job name.
2.
Cancellation Process: When a flow run is canceled, Prefect attempts to terminate the associated infrastructure. However, there are a few caveats:
- If the infrastructure does not support cancellation, the process won't work.
- If the infrastructure cannot be found or has already been killed, the flow run is marked as canceled.
- If the worker encounters an error during cancellation, it may retry or another worker may attempt the cancellation.
3.
Command Line and UI: You can cancel a flow run using the Prefect CLI with
prefect flow-run cancel <flow-run-id>
or through the Prefect UI.
For more details, you can refer to the following documentation:
-
Cancellation Process
-
Kubernetes Infrastructure
It seems that Prefect already handles the termination of Kubernetes Jobs when a run is canceled, but you should ensure your infrastructure supports this feature. If you have specific configurations or custom setups, you may need to verify compatibility.