https://prefect.io logo
l

Lee Mendelowitz

07/06/2023, 5:27 PM
Is there a best practice for how to retry flow runs that crash? Our specific use case is for retrying flows that run as ECSTask’s on Fargate Spot.
1
b

Bianca Hoch

07/06/2023, 6:11 PM
Hi Lee, you could try creating an automation in the UI which is triggered whenever a run for a particular flow enters a Crashed state. The action in response to the trigger would be to run a deployment (in this case I think
Infer Deployment
would suffice).
Screenshot 2023-07-06 at 2.12.42 PM.png,Screenshot 2023-07-06 at 2.12.48 PM.png
l

Lee Mendelowitz

07/06/2023, 7:15 PM
Cool thanks Bianca
🚀 1
1
It looks like this isn’t working for us because it doesn’t resubmit the flow run with the parameters set.
b

Bianca Hoch

07/10/2023, 3:34 PM
Hmm. Maybe using state change hooks will be the best path forward. There's an
on_crashed
hook that will execute code in response to the flow run entering a
Crashed
state. You could write a hook which calls run_deployment(), and pass in the desired parameters to be used for the new flow run.
With that being said, I think it'd be a cool feature request to have automations pass flow run metadata or context to a flow run if the "Run Deployment" option is selected. To my understanding this option isn't available right now. I'll raise this to the team!
l

Lee Mendelowitz

07/10/2023, 5:59 PM
Awesome that would be great