https://prefect.io logo
Join Slack
Powered by
Hello Everyone, I am running prefect server on K8s...
# prefect-server
a
Hello Everyone, I am running prefect server on K8s. I am checkpointing certain tasks but not all. Seems like when the flow breaks and i restart it, the flow is running from the very beginning rather than the point where code broke. How can make sure the flow restarts from the task where it failed #prefect-server
✅ 1
k
How are you restarting it?
a
by clicking on restart button on the dashboard
k
That’s a bit weird. How did you configure results?
a
i am using S3 results and mentioned it in flow itself.
then for each task , i added checkpoint True/False values
So basically lets say i have logs for 4 tasks task1---> task2--->task3---->task4 = flow
i am checkpointing 
task 3
and persisting it in S3 and the code breaks at 
task 4
, when i press the restart button , its says 
null restarted the flow
and all the logs from very start show up again
k
Did you attach the result in a separate registration script or is it in the same file?
a
result is in the same file
k
Do you see the result written out in S3?
a
yes
Copy code
with Flow("Advance Flow 2", state_handlers=[flow_alert], result  = S3Result(bucket='bucket_name', location ="prefect_logs/results/"+"{date:%Y-%m-%d}/{task_name}"+"_{date:%H}.prefect")) as f:
k
I think task level results might be better for this to guarantee the file is tied to the task run
Yeah I think that’s what I would try first
a
ok so @Kevin Kho could you please help me with one small question , lets say i have a flow like this.
Copy code
task1---> task2--->task3---->task4 = flow
question-1) what will happen if i persist output of 
task3
and code breaks at 
task4
and i restart the flow from dashboard. question-2) what will happen if i persist output of 
task3
and code breaks at `task2`and i restart the flow from dashboard
ok let me try using task level results approach as well.
k
1. Restart should use task3 output to pass to task4 2. Assuming task2 is a dependency of task3, task3 won’t even run here by default?
a
So for question 2 , restarting will actually run the whole flow from very start ?
k
No restart shouldn’t run already successful tasks so it should start from task2 and not run task1
You may get a log like
Copy code
Starting task1...
but it should just end immediately and not run
a
ohhh got it, yes i am getting logs for each tasks
so does that mean in my original question logs for each tasks will appear but they are not running actually
k
Yeah because it will just already see it’s in a success state
a
ok great understood it, one last thing Kevin for the question-2 i mentioned above you said the 
task2
will run and not 
task1
as it already ran successfully but knowing that i only persist 
task3
result and the dependency order of the flow is
Copy code
task1---> task2--->task3---->task4
Since 
task1
is not persisted shouldn't it run again?
k
No it will not run again. And task 2 might break on restart because it doesn’t know where to retrieve the old task 1 output.
If you want it to run again, you can just start a new flow run, and use 
caching
along with the results. Docs here and this won’t re-run the task
a
Thats super amazing @Kevin Kho, So summarising what you just told and what i undertood If i persist a task and flow fails at next downstream task, flow will restart from downstream task where it broke and prefect is capable to retrieve the persisted result and use it again on restart.
So do you recommend persisting all the tasks output?
k
Yes if expensive and you need restarts. If you don’t need it also as a downstream input, no need to persist
a
thank you Kevin🙏 😃
k
Of course!
8 Views
Open in Slack
PreviousNext
Powered by