@Marvin in prefect 2, if I cancel a flow, will the termination of infrastructure be blocked by a running task?

In Prefect 2.x, canceling a flow will not necessarily be blocked by a running task. The cancellation process is generally robust, but there are some scenarios where issues might occur: 1. Infrastructure Constraint: - If the infrastructure block for the flow run has been removed or altered, it might prevent proper cancellation. - If the infrastructure block does not support cancellation, this will also affect the process. 2. Worker Scope: - If the worker handling the flow run cannot match the scope for cancelling (perhaps due to infrastructure not being found or already killed), the flow run will be marked as cancelled without actually enforcing cancellation. 3. Error Handling: - If there is a missing infrastructure PID or an unexpected error occurs, it might prevent proper enforcement of the cancellation. However, workers make attempts to cancel the flow, and another worker might take over this task if the first one fails. In summary, while running tasks may not directly block the termination of infrastructure during a flow cancellation, certain infrastructural or worker-related issues could potentially cause cancellation not to take immediate effect. For more detailed information, you can refer to this link: Prefect Infrastructure. If you have further questions or need to know more about a specific case, feel free to ask!

@Marvin I cancel a job from the dashboard and it shows that the state has been moved to cancelled, but I still see logs in the pod printing progress being completed

When you cancel a flow from the Prefect dashboard and the UI shows that the state has moved to "Cancelled," but you still see logs in the pod indicating progress, it can be due to a few reasons related to the nature of flow cancellation in Kubernetes environments: 1. Delayed Cancellation on Kubernetes: The cancellation command might be executed, marking the flow as "Cancelled" in Prefect's system. However, the underlying Kubernetes pods (or other infrastructure) might still be running tasks. The infrastructure might not instantly shut down the tasks, leading to logs appearing even after cancellation. 2. Infrastructure Cleanup: There could be a disconnect where the Prefect flow pod is terminated, but the associated Kubernetes pods (like Dask Scheduler or Worker pods) continue to exist, as seen in a known issue on GitHub (Issue #12976). The flow pod might not send proper termination signals, leaving other infrastructure components orphaned. 3. Enhanced Cancellation: Prefect is working on improving cancellation issues. You can try the experimental enhanced cancellation feature by setting

PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION=True

. This might address some of the cancellation enforcement issues by providing better control over the ongoing tasks. For more information, you can refer to the source: - Prefect Flow Cancellation and Infrastructure If the issue persists, consider filing a detailed report with Prefect's support or on their GitHub repository for a more tailored solution.