Hi there! I have a flow which has been failing at an increasing frequency recently. The actual task logic is always completed, but then I get

Finished in state Failed('Flow run encountered an exception. MissingResult: State data is missing. Typically, this occurs when result persistence is disabled and the state has been retrieved from the API.\n')

at the end. Manually retrying the flow succeeds, but I do have

retries=3

in the

flow

decorator and the flow doesn't appear to be retried

We’re chatting about (possibly) the same thing in https://prefect-community.slack.com/archives/CL09KU1K7/p1673031232195399

so i was able to figure out what caused this issue here. we basically have automation that will cancel runs that have been in

pending

for more than 60 seconds because we assume that if its in pending for more than 60 seconds the container the process is running in has been terminated. the reason for this is complicated to explain, so i wont do it here. when we either set the state of a run to

cancelled

or

scheduled

(awaiting retry) while its pending or running, then the run goes into

running

, the flow run will spit out the error above when it tries to compute state next

!!!

Ah so this is very similar in that the flow run is not in a running state so the orchestrator denies the task’s attempt to run.