Is there a best practice for how to retry flow runs that crash? Our specific use case is for retryin...

Lee Mendelowitz

07/06/2023, 5:27 PM

Is there a best practice for how to retry flow runs that crash? Our specific use case is for retrying flows that run as ECSTask’s on Fargate Spot.

✅ 1

Bianca Hoch

07/06/2023, 6:11 PM

Hi Lee, you could try creating an automation in the UI which is triggered whenever a run for a particular flow enters a Crashed state. The action in response to the trigger would be to run a deployment (in this case I think

Infer Deployment

would suffice).

Bianca Hoch

07/06/2023, 6:13 PM

Lee Mendelowitz

07/06/2023, 7:15 PM

Cool thanks Bianca

✨ 1

🚀 1

Lee Mendelowitz

07/09/2023, 8:52 PM

It looks like this isn’t working for us because it doesn’t resubmit the flow run with the parameters set.

Bianca Hoch

07/10/2023, 3:34 PM

Hmm. Maybe using state change hooks will be the best path forward. There's an

on_crashed

hook that will execute code in response to the flow run entering a

Crashed

state. You could write a hook which calls run_deployment(), and pass in the desired parameters to be used for the new flow run.

Bianca Hoch

07/10/2023, 3:36 PM

With that being said, I think it'd be a cool feature request to have automations pass flow run metadata or context to a flow run if the "Run Deployment" option is selected. To my understanding this option isn't available right now. I'll raise this to the team!

Lee Mendelowitz

07/10/2023, 5:59 PM

Awesome that would be great

14 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.