@Marvin Once the flow run is crashed due to the EKS pod size is too small, can it continue to run the on_crashed post hook

Oops, it seems I had a tiny oversight and forgot to specify the search queries. Let me start those searches again, but with the queries included this time. Hang tight! When dealing with a flow run that crashes, such as due to a small EKS pod size, Prefect provides state change hooks to handle such scenarios. Both Prefect 2.x and 3.x offer an

on_crashed

state change hook which is explicitly designed to handle when a flow run enters a "Crashed" state. This enables you to execute custom logic, such as notifications or logging, even if the main execution fails. Here's what I found: 1. Prefect 3.x: Prefect 3.x supports various state change hooks, including the

on_crashed

one, allowing you to define custom behaviors when a flow runs into a Crashed state. This can be a very useful feature if you want to handle the aftermath of an unexpected crash programmatically. You can find detailed information on managing states in the Prefect 3.x documentation. 2. Prefect 2.x: Similarly, Prefect 2.x provides state change hooks such as

on_crashed

to react when a flow run crashes. You can set up such hooks to perform specific actions or notifications automatically. Here's a link to the relevant section in the Prefect 2.x documentation. So next time your EKS pod decides to take a break and causes your flow to crash, you've got

on_crashed

to help manage the situation like a pro!

@Marvin I set up the

on_crashed

state change hook, but when my workflow runs into the crashed status, it didn't run that hook at all. Is that because the EKS pod is dead?

It looks like I've run into a bit of trouble accessing GitHub issues at the moment—robots and authentication, you know. While I can't directly search GitHub issues, I can offer some insights that might help: 1. Execution Environment: When an EKS pod crashes, it’s possible that the execution context needed for running the

on_crashed

hooks is also affected. If the pod is completely unresponsive, it may not be able to execute any further logic contained within the hook since the environment it runs in (the pod) is no longer operational. 2. Check Prefect Logs: Ensure that Prefect's logging does not indicate any errors that would prevent the state hook from executing, such as a lack of connectivity to the Prefect server or the Prefect agent being unable to report the state change. 3. Deployment Configuration: Make sure the Prefect deployment is correctly set up to use state change hooks on your infrastructure, and the environment variables or configurations are not missing. 4. Redundancy Setup: Consider setting up a more robust retry or redundancy mechanism to handle pod failures, such as using more stable nodes or setting up automatic pod restarts in case of failures. You might want to check Prefect's discussion forums or community Slack channels for similar experiences, as other users might have encountered and solved similar issues. If you're able to, connecting with the Prefect community can be a great way to get specific debugging advice!