Hello Everyone I am running prefect server on K8s I am check

Question

Hello Everyone I am running prefect server on K8s I am checkpointing certain tasks but not all Seems like when the flow breaks and i restart it the flow is running from the very beginning rather than the point where code broke How can make sure the flow restarts from the task where it failed < C014Z8DPDSR|prefect server>

Kevin Kho · Answer

How are you restarting it

ash · Answer

by clicking on restart button on the dashboard

Kevin Kho · Answer

That s a bit weird How did you configure results

ash · Answer

i am using S3 results and mentioned it in flow itself

ash · Answer

then for each task i added checkpoint True False values

ash · Answer

So basically lets say i have logs for 4 tasks task1 gt task2 gt task3 gt task4 = flow

ash · Answer

i am checkpointing `task 3` and persisting it in S3 and the code breaks at `task 4` when i press the restart button its says `null restarted the flow` and all the logs from very start show up again

Kevin Kho · Answer

Did you attach the result in a separate registration script or is it in the same file

ash · Answer

result is in the same file

Kevin Kho · Answer

Do you see the result written out in S3

ash · Answer

yes

ash · Answer

```with Flow Advance Flow 2 state handlers= flow alert result = S3Result bucket= bucket name location = prefect logs results + date Y m d task name + date H prefect as f ```

Kevin Kho · Answer

I think task level results might be better for this to guarantee the file is tied to the task run

Kevin Kho · Answer

Yeah I think that s what I would try first

ash · Answer

ok so < Kevin Kho> could you please help me with one small question lets say i have a flow like this ```task1 gt task2 gt task3 gt task4 = flow``` question 1 what will happen if i persist output of `task3` and code breaks at `task4` and i restart the flow from dashboard question 2 what will happen if i persist output of `task3` and code breaks at `task2`and i restart the flow from dashboard

ash · Answer

ok let me try using task level results approach as well

Kevin Kho · Answer

1 Restart should use task3 output to pass to task4 2 Assuming task2 is a dependency of task3 task3 won t even run here by default

ash · Answer

So for question 2 restarting will actually run the whole flow from very start

Kevin Kho · Answer

No restart shouldn t run already successful tasks so it should start from task2 and not run task1

Kevin Kho · Answer

You may get a log like ```Starting task1 ``` but it should just end immediately and not run

ash · Answer

ohhh got it yes i am getting logs for each tasks

ash · Answer

so does that mean in my original question logs for each tasks will appear but they are not running actually

Kevin Kho · Answer

Yeah because it will just already see it s in a success state

ash · Answer

ok great understood it one last thing Kevin for the question 2 i mentioned above you said the `task2` will run and not `task1` as it already ran successfully but knowing that i only persist `task3` result and the dependency order of the flow is ```task1 gt task2 gt task3 gt task4```

ash · Answer

Since `task1` is not persisted shouldn t it run again

Kevin Kho · Answer

No it will not run again And task 2 might break on restart because it doesn t know where to retrieve the old task 1 output

Kevin Kho · Answer

If you want it to run again you can just start a new flow run and use `caching` along with the results <https docs prefect io core concepts persistence html output caching|Docs here> and this won t re run the task

ash · Answer

Thats super amazing < Kevin Kho> So summarising what you just told and what i undertood If i persist a task and flow fails at next downstream task flow will restart from downstream task where it broke and prefect is capable to retrieve the persisted result and use it again on restart

ash · Answer

So do you recommend persisting all the tasks output

Kevin Kho · Answer

Yes if expensive and you need restarts If you don t need it also as a downstream input no need to persist

ash · Answer

thank you Kevin pray smiley

Kevin Kho · Answer

Of course