Sorry, lots of questions because Prefect seems to be erroring much more than usual. I have an automation that cancels and then retries a flow that has been waiting for 30 mins. For the last few days I am being notified with an error of:

State message: Submission failed. IndexError: list index out of range

cc: @Will Raphaelson in case this is good info for you.

Thanks Ben - I think this is, as currently spelled “against the rules”. A cancelled state is terminal, so state transitions that are proposed from cancelled to another state are rejected. That said, I think we can improve the error message here for sure. I believe that this can / should be accomplished without automations for individual flow runs (via flow retries) but automations could help here in terms of setting something like a global retry policy. We could consider surfacing a retry action type for this case. I’ll file a tracking issue internally. Going to tag in my orchestration expert colleagues who may have perspective as well, thanks.

If you see my question two above, this cant be done with the decorator.

ah yes sorry, I see that now. yeah we agree this is a good use case for automations. I dont have a precise timeline yet but I want to get this done.

awesome thanks Will. Getting heaps of these recently and I believe they are Prefect issues, so would be great to have a way to handle it.

Yeah, in the interim unfortunately I dont think we have an elegant way to support this retry after x time in y state. Would it work to have two actions fire - one to cancel the inferred flow run and one to kick off a new one? I know its not ideal but it might be a bandaid until we can introduce retry action types?

I think we aren't on the same page - why would this not work?

Ahh okay i think i understand now. It is my expectation that the configuration in that screenshot would in fact work. And just to clarify you’ve taken the retries argument off the flow right? If so, I think I could use a github issue to dig in more, would you provide the URL, relevant stack traces, etc in an issue in the prefect repo? I’ll try to reproduce locally as well.

it is for all flows - so I cant confirm I have taken the decorator off, because it is likely that some flows will have it. The retry decorator will have zero impact here because the flows are not failing - all we are checking for is if they are in pending or late. When the first automation action runs it will cancel it - the retry decorator only triggers for "FAILED" flows.

that would surprise me but its worth a try, im poking around now. where exactly are you receiving that indexerror, on the event for the automation.failed right?

I get it on the notification

okay i thought the full automation was failing including the kickoff of the second flow run.