Hello everyone, retrying a simple flow with subflows with no result storage causes error (MissingRes...

Deceivious

07/27/2023, 8:14 AM

Hello everyone, retrying a simple flow with subflows with no result storage causes error (MissingResult). Is this expected?

✅ 1

Deceivious

07/27/2023, 8:16 AM

Replication code

Copy code

import prefect
from prefect import flow
from prefect.deployments.deployments import Deployment

@flow
def success():
    prefect.get_run_logger().info("a")


@flow
def fail():
    prefect.get_run_logger().info("b")


@flow
def main_flow():
    prefect.get_run_logger().info("c")
    success()
    fail()


if __name__ == "__main__":
    Deployment.build_from_flow(flow=main_flow, name="test_flow", apply=True)

Deceivious

07/27/2023, 8:17 AM

Create a flow run using the deployment page - run it on an agent. Once the flow run is completed, retry the same flow run using the UI. The retry should fail with a Missing Result error.

Deceivious

07/28/2023, 9:54 AM

Reping for attentions

Christopher Boyd

07/28/2023, 12:51 PM

Hey Deceivious, result_persistence is turned off by default, and there is no code in your repro that toggles it on

Christopher Boyd

07/28/2023, 12:51 PM

https://docs.prefect.io/2.11.0/concepts/results/#toggling-persistence

Deceivious

07/28/2023, 12:53 PM

@Christopher Boyd But my code does not require the result.

Deceivious

07/28/2023, 12:54 PM

Copy code

The persist_result option defaults to a null value, which will automatically enable persistence if it is needed for a Prefect feature used by the flow or task.

Also according to docs

Christopher Boyd

07/28/2023, 12:55 PM

Maybe I misunderstand your problem : • There is no result persistence on the flow • Default result persistence is disable • You get a MissingResult error when you retry

Christopher Boyd

07/28/2023, 12:55 PM

what is the expected outcome here?

Christopher Boyd

07/28/2023, 12:55 PM

retrys require results

Deceivious

07/28/2023, 12:57 PM

I would expect the result to be retrieved only if it is being used.

Christopher Boyd

07/28/2023, 12:57 PM

What’s the point of retrying then, instead of just running a new flow?

Christopher Boyd

07/28/2023, 12:58 PM

Retrying is implicitly saying use results

Deceivious

07/28/2023, 12:59 PM

Say one of my sub flows failed. I have 17 of them . Id want to skip the 16 of them and retry the 1 that has failed. My sub flows do not return anything. I would expect the 16 subflows; state to be checked ie Completed and the ones that Failed would be retried.

Christopher Boyd

07/28/2023, 1:00 PM

Are results persisted for any of them? Parent flow or subflows?

Deceivious

07/28/2023, 1:01 PM

No result persisted for any of them. All the results are None.

Christopher Boyd

07/28/2023, 1:02 PM

I’ll check with the team and see if there is any other feedback - this seems like expected behavior to me , but I could be wrong

Deceivious

07/28/2023, 1:03 PM

Thanks @Christopher Boyd

Deceivious

07/28/2023, 1:07 PM

My concern with this approach is: 1. I am using a distributed agent ie. multiple agents in multiple machines on the cloud. 2. My code has no returns ie all of them return

None

3. Id say it would be weird if I had to allocate a shared remote storage infrastructure to store those `None`s in esp. as the results are not being used at all. The status of the subflows are in Prefect database.

Deceivious

07/31/2023, 8:12 AM

Hi @Christopher Boyd, Any update on this?

Christopher Boyd

07/31/2023, 12:23 PM

I checked in with the team - the expectation here is that this is as expected. If you want something to be retryable, results need to be persisted

Deceivious

07/31/2023, 1:05 PM

Thanks.

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.