@Marvin In what cases or scenarios will an on_crashed hook run or fail to run for a flow? We're on prefect 2.x

In Prefect 2.x, the

on_crashed

hook is designed to run when a flow enters a "crashed" state. This typically happens if the flow's execution is abruptly interrupted, such as due to infrastructure issues or unexpected termination. Here are some scenarios regarding its execution: 1. When it Runs: - The

on_crashed

hook will execute when the flow run crashes unexpectedly. This could be due to unhandled exceptions, external interruptions, or failures in infrastructure that cause the process to be forcibly terminated. - It's commonly used for cleanup purposes, like deleting resources that were instantiated during the flow run but should not persist after a crash. 2. When it Might Not Run: - The hook might not execute if the process running the flow is terminated so abruptly that even the crash handling within Prefect cannot catch it. - Network issues or disconnection from Prefect's backend might also prevent the

on_crashed

state from being recorded and the hook from executing. To effectively use the

on_crashed

hook, you should ensure that your infrastructure setup allows Prefect to capture and respond to crashes reliably. You can find more details and examples in the Prefect documentation here: State Change Hooks in Prefect 2.x.

@Marvin Would an Out of Memory (OOM) error cause the hook to not run?

An Out of Memory (OOM) error can lead to situations where an

on_crashed

hook might not run. Here's how it can affect execution: 1. Potential for Hook Not to Run: - OOM errors cause the system to abruptly terminate processes to free up memory. When a process is killed in such a harsh manner, it might not allow Prefect's execution environment to transition the flow into a "crashed" state in time for the hook to execute. This is because the necessary resources (memory in this case) to handle such state changes and cleanups are critically limited. 2. Ensuring Hook Execution: - To better handle such scenarios, you can implement memory monitoring and proactive management strategies in your flows to catch potential memory issues before they escalate to OOM errors. - Consider specifying memory constraints and optimizing resource usage within your flows, or using Prefect's infrastructure options to restart or handle flows in cases of high memory usage. In summary, due to the abrupt nature of OOM errors, the

on_crashed

hook might not run, depending on when and how the system decides to terminate the process.

@Marvin if a flow or task is stuck in a non-terminal state, is there a query I can perform to check that the infrastructure no longer exists?

In Prefect 2.x, if a flow or task is stuck in a non-terminal state and you suspect that the associated infrastructure is no longer existing, you can perform a few checks and queries: 1. Logs: Check the logs for any indicators that the infrastructure was terminated or that there were connectivity issues. Infrastructure-related logs might provide clues if the environment was disrupted. Prefect agent or worker logs, in particular, could be informative. 2. Infrastructure Block: Ensure that the infrastructure block, which defines the execution environment, is properly configured. Review the documentation on Prefect Infrastructure to confirm your setup is correct. 3. Programmatic Verification: If possible, you may write a custom task in your flow that ping the infrastructure service (like AWS, GCP, Docker, etc.) to confirm whether the expected resources (VM, service, etc.) are still active. 4. Prefect API: Though there might not be a direct API for querying disappeared infrastructure, you can interact with the platform resources through tasks or external scripts that can help verify the infrastructure's current status. If the infrastructure was inadvertently terminated, the flow might be unable to transition into another state, leading to a stuck scenario. Consider implementing monitoring and alerting systems to notify you of such issues promptly.