@Marvin when a prefect flow crashes or is cancelled, the children tasks and subflows remain running, what can I do

When a Prefect flow crashes or is canceled, the child tasks and subflows might not terminate automatically. This behavior can be influenced by the infrastructure and configuration used for running the flow. Here are some insights and steps you can take: 1. Cancellation Support: Ensure that the infrastructure used for running your flows supports cancellation. Sometimes, the infrastructure might not support this feature, leading to child tasks continuing to run. More details can be found in the Prefect documentation on cancellation. 2. Flow Engine: The Prefect flow engine has a method

cancel_all_tasks

that can be invoked to attempt canceling all running tasks in a flow. This is part of the flow's task runner. You can explore using this method programmatically if you have access to the flow's task runner. For more details, you can look at the flow engine source code. 3. Handling Crashes: When handling crashes, ensure that the state is updated correctly to reflect the crash, which can help in managing subsequent task cancellations. The flow engine in Prefect has mechanisms to handle and log crashes which might be useful for debugging and managing task states. 4. Infrastructure Identifiers: Make sure the identifiers for your infrastructure (like Docker container IDs or Kubernetes job names) are correctly set up to ensure proper cancellation. In summary, check your infrastructure's cancellation support, consider using the

cancel_all_tasks

method if applicable, and ensure proper state handling during crashes. If you need further customization, you might have to dive into the flow engine's internals or adjust your infrastructure setup.

@Marvin do tasks of crashed flows get cancelled automatically

When a Prefect flow crashes, tasks within that flow do not automatically get canceled. The behavior largely depends on the infrastructure and configuration: 1. Infrastructure Support: If the infrastructure supports cancellation and the identifiers are correctly scoped, tasks might be canceled by Prefect workers. However, if the infrastructure does not support cancellation, tasks will continue to run even if the flow is marked as canceled. 2. Flow Engine Handling: The Prefect flow engine has mechanisms to handle crashes and update the state accordingly, but it does not guarantee automatic task cancellation. The tasks might continue unless explicitly canceled through other means. For more details, you can refer to the Prefect documentation on cancellation and the flow engine source code for handling flow crashes. In practice, you might need to implement additional logic to manage task cancellations in the event of a flow crash, especially if using infrastructure that doesn't inherently support task cancellation.