Hi all, I’ve set up an Automation to re-run flows that crash. This nicely catches random crashes but goes haywire when an error inherent to the flow causes crashes — the flow will re-run and crash ad infinitum. Is there any way to limit the number of times an Automation is triggered? In this case I’d like Prefect to try re-running only once, then stand down.

I don’t think so, but this is good feedback - I think something like a

limit

could be good here

Digging around I can find an alternative fix, which is to put in place another Automation which stops the work queue after a set number of failures/crashes, as per here https://github.com/anna-geller/prefect-cloud-automations/blob/main/flow_runs/pause_work_queue_after_3_failures.py

yeah this is a good feedback thank you. I see two potential approaches with the current features. • Use retries in the flow decorator. This will only retry up to x times and then it will fail terminally. could be annoying to add this to all relevant flows. • Set up a another automation as a watcher, as you wrote above.

Thanks for the quick reply both, I appreciate your taking this into account for the future

oh thats a good callout - let me see if there is a technical reason we dont support retries on crashed states, if we did, we’d get get this counting “for free” and would be useful.

You do support retries on crashed states! You just don’t support stopping them ^_^

wouldn’t @flow(retries=5) get you where you need to go?

ah OK I misunderstood — I’m not sure whether flow retries support crashed states or not. I thought you were referring to automations.

yeah let me know how that goes. im almost positive that this will work well, and if it doesn’t we should build support for it and it shouldnt be too hard.

retries

as a flow parameter does not appear to trigger (or limit) retries on crashes

thanks for trying that robert. if you want to open a github issue for supporting crashed states on flow retries in our github repo we can discuss and prioritize there!

Sounds good, will do

thanks!