https://prefect.io logo
Title
a

ash

07/05/2022, 4:45 PM
Hello Everyone, I am running prefect server on K8s. I am checkpointing certain tasks but not all. Seems like when the flow breaks and i restart it, the flow is running from the very beginning rather than the point where code broke. How can make sure the flow restarts from the task where it failed #prefect-server
1
k

Kevin Kho

07/05/2022, 4:45 PM
How are you restarting it?
a

ash

07/05/2022, 4:46 PM
by clicking on restart button on the dashboard
k

Kevin Kho

07/05/2022, 4:52 PM
That’s a bit weird. How did you configure results?
a

ash

07/05/2022, 4:53 PM
i am using S3 results and mentioned it in flow itself.
then for each task , i added checkpoint True/False values
So basically lets say i have logs for 4 tasks task1---> task2--->task3---->task4 = flow
i am checkpointing
task 3
and persisting it in S3 and the code breaks at
task 4
, when i press the restart button , its says
null restarted the flow
and all the logs from very start show up again
k

Kevin Kho

07/05/2022, 4:56 PM
Did you attach the result in a separate registration script or is it in the same file?
a

ash

07/05/2022, 4:56 PM
result is in the same file
k

Kevin Kho

07/05/2022, 4:57 PM
Do you see the result written out in S3?
a

ash

07/05/2022, 4:57 PM
yes
with Flow("Advance Flow 2", state_handlers=[flow_alert], result  = S3Result(bucket='bucket_name', location ="prefect_logs/results/"+"{date:%Y-%m-%d}/{task_name}"+"_{date:%H}.prefect")) as f:
k

Kevin Kho

07/05/2022, 4:58 PM
I think task level results might be better for this to guarantee the file is tied to the task run
Yeah I think that’s what I would try first
a

ash

07/05/2022, 5:02 PM
ok so @Kevin Kho could you please help me with one small question , lets say i have a flow like this.
task1---> task2--->task3---->task4 = flow
question-1) what will happen if i persist output of
task3
and code breaks at
task4
and i restart the flow from dashboard. question-2) what will happen if i persist output of
task3
and code breaks at `task2`and i restart the flow from dashboard
ok let me try using task level results approach as well.
k

Kevin Kho

07/05/2022, 5:03 PM
1. Restart should use task3 output to pass to task4 2. Assuming task2 is a dependency of task3, task3 won’t even run here by default?
a

ash

07/05/2022, 5:04 PM
So for question 2 , restarting will actually run the whole flow from very start ?
k

Kevin Kho

07/05/2022, 5:04 PM
No restart shouldn’t run already successful tasks so it should start from task2 and not run task1
You may get a log like
Starting task1...
but it should just end immediately and not run
a

ash

07/05/2022, 5:05 PM
ohhh got it, yes i am getting logs for each tasks
so does that mean in my original question logs for each tasks will appear but they are not running actually
k

Kevin Kho

07/05/2022, 5:06 PM
Yeah because it will just already see it’s in a success state
a

ash

07/05/2022, 5:09 PM
ok great understood it, one last thing Kevin for the question-2 i mentioned above you said the
task2
will run and not
task1
as it already ran successfully but knowing that i only persist
task3
result and the dependency order of the flow is
task1---> task2--->task3---->task4
Since
task1
is not persisted shouldn't it run again?
k

Kevin Kho

07/05/2022, 5:10 PM
No it will not run again. And task 2 might break on restart because it doesn’t know where to retrieve the old task 1 output.
If you want it to run again, you can just start a new flow run, and use
caching
along with the results. Docs here and this won’t re-run the task
a

ash

07/05/2022, 5:15 PM
Thats super amazing @Kevin Kho, So summarising what you just told and what i undertood If i persist a task and flow fails at next downstream task, flow will restart from downstream task where it broke and prefect is capable to retrieve the persisted result and use it again on restart.
So do you recommend persisting all the tasks output?
k

Kevin Kho

07/05/2022, 5:17 PM
Yes if expensive and you need restarts. If you don’t need it also as a downstream input, no need to persist
a

ash

07/05/2022, 5:21 PM
thank you Kevin🙏 😃
k

Kevin Kho

07/05/2022, 5:22 PM
Of course!