Hi experts, we are currently having problem with retrying failed flow starting from the point of the failure. We created a master control flow to using

StartFlowRun

to invoke individual flows in their defined orders. All the flows run in their own docker container. When master flow execution flow fails due to specific failed execution, it was not working. I read in docs and said that docker run does not yet support retaining results/inputs/outputs. I read a thread here by @Kevin Kho and seemed to have suggested that PrefectResult may be a workaround if we don't care about the data we push to Prefect Cloud. https://prefect-community.slack.com/archives/CL09KU1K7/p1622645858327700 I am not exactly sure how PrefectResult can be used in our scenario. Please help!

Hi @liren zhang, I actually think this behavior might be more seamless if you use the new KV Store. You can persist a value for later retrieval. How big is the data being passed around?

Hi @Kevin Kho KV store seems like a good solution. I want to ask about the a more fundamental question about task retry after the entire flow end up in failed state due to a task failure inside of the flow. If the failed task has a input from the previous task result, how does Prefect actually

retry

using the previous input value? How does this work behind the scene.

Return values of tasks are persisted as

Results

if checkpointing is on, which it is for Prefect Cloud. When a task is retried, it retrieves the upstream results needed.

so if I use PrefectResult to store the task inputs/outputs, we will be ok even we run flows in docker container?

I believe so. The only note here is that Prefect by default does not see any of your data. Using the PrefectResult means we see it. It also might not work for large DataFrames because we’re storing that in our database

So when I click on

retry

button from Prefect Cloud UI, how exactly Prefect bring back the previous inputs and continue the execution of the past flow run?

S3Result* Storage is where for is stored. But yes configure the task with the Result

@task(result = S3Result(...))

and then when a re-run happens, it knows where to retrieve the output of upstream tasks as it continues the Flow.

I will try this out. Thanks

Oh I read your original question though. I have been talking in the context of a flow. For the

StartFlowRun

task, the result is not the location of data unfortunately. This will be fixed later on but for now, it’s better to manually persist and retrieve for data dependencies between sub flow runs

I am exactly clear on how a downstream flow retrieve the previous run input when I manually click on the retry button from the prefect UI. Do you have some sort of examples that demonstrate this

Yeah I’ll make one for you. This is without StartFlowRun right?

I was hope for an example with StartFlowRun as that is what use case I am dealing with.