Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Is there a best practice for how to retry flow runs that crash? Our specific use case is for retrying flows that run as ECSTask’s on Fargate Spot.

Hi Lee, you could try creating an automation in the UI which is triggered whenever a run for a particular flow enters a Crashed state. The action in response to the trigger would be to run a deployment (in this case I think `Infer Deployment` would suffice).

Screenshot 2023-07-06 at 2.12.48 PM.png

Screenshot 2023-07-06 at 2.12.42 PM.png

It looks like this isn’t working for us because it doesn’t resubmit the flow run with the parameters set.

Hmm. Maybe using <https://docs.prefect.io/2.10.20/concepts/states/?h=triggers#state-change-hooks|state change hooks> will be the best path forward. There's an `on_crashed` hook that will execute code in response to the flow run entering a `Crashed` state. You could write a hook which calls <https://docs.prefect.io/2.10.20/api-ref/prefect/deployments/deployments/?h=run_deployment#prefect.deployments.deployments.run_deployment|run_deployment()>, and pass in the desired parameters to be used for the new flow run.

With that being said, I think it'd be a cool feature request to have automations pass flow run metadata or context to a flow run if the "Run Deployment" option is selected. To my understanding this option isn't available right now. I'll raise this to the team!