Hi! I have a flow running on preemptible nodes on ...
# ask-communityc
Cab Maddux
04/16/2021, 7:49 PMHi! I have a flow running on preemptible nodes on GKE, which looks to have been preempted and subsequently caught by the Zombie Killer. My flow was then marked as failed but I expected the Lazarus process to pick it up and restart the flow after 10 minutes.
As seen in the screenshot the Zombie killer fails the flow at 13:48 and then nothing until I manually restart the flow via the Prefect Cloud UI at 14:45 about an hour later. I confirmed that Lazarus is enabled for this flow. Is there anything else I need to do to have Lazarus pick this up?
Cab Maddux
04/16/2021, 7:52 PMHere are the same logs at the flow level rather than the task level in case useful
k
Kevin Kho
04/16/2021, 8:00 PM
Hi @Cab Maddux! I’ll look into this for you. Can you tell me what Agent you are using?
c
Cab Maddux
04/16/2021, 8:03 PMThanks Kevin, we're using the k8s agent
k
Kevin Kho
04/16/2021, 8:07 PM
I may have to get back to you on this on Monday, but I won’t forget
👍 1
Kevin Kho
04/16/2021, 8:37 PM
Hi @Cab Maddux, Lazarus will not pick up failed flows. Lazarus picks up a flow run for two reasons:
1. its in a submitted state for a long time
2. its in a running state and stops sending a heartbeat for some period of time
c
Cab Maddux
04/16/2021, 8:40 PMThanks @Kevin Kho yeah, my thought is that this falls under case #2, right?
(I've attached where in the docs it looks like in this case the Lazarus process should be restarting this flow - due to a No Heartbeat error)
Cab Maddux
04/16/2021, 8:42 PMUnless I've misunderstood something here
k
Kevin Kho
04/16/2021, 8:43 PM
Yeah, I’ll need to get some clarification from some team members that are already out. Will get back on Monday for sure.
c
Cab Maddux
04/16/2021, 8:46 PMNo worries, thanks!
n
nicholas
04/19/2021, 3:03 PM
Hi @Cab Maddux - can you confirm that your flow was in a
RunningFailednicholas
04/19/2021, 3:03 PM
(I'm having a little trouble deciphering that from the logs you posted, apologies if you've already checked this!)
c
Cab Maddux
04/19/2021, 3:10 PMHi @nicholas yeah I think the flow was in a running state when the zombie task marked as failed. If you look at the first screenshot in this thread it looks like the
basecall_fast5_fileCab Maddux
04/19/2021, 3:11 PMEverything I can tell seems to indicate the flow is in a running state when the zombie task comes in
n
nicholas
04/19/2021, 3:26 PM
Interesting, let me see if I can recreate this
nicholas
04/19/2021, 4:18 PM
Ah I apologize @Cab Maddux - I misspoke earlier and the behavior you saw was indeed correct. The Lazarus service doesn't re-submit failed task runs, instead it handles re-scheduling (or failing, after 10 tries) flow runs that are in running or submitted states that have no pending or running tasks. Without a fuller picture of the run I can't tell exactly why your flow run was marked as failed, but if you had other task runs pending/running at the time, Lazarus wouldn't pick your run and the run could be failed for other reasons (including because your task was failed by the zombie killer)
c
Cab Maddux
04/20/2021, 1:05 PMHi @nicholas thanks so much, I'll do some additional digging on my end. Just to clarify, if we have a flow running on k8s fine and the underlying node gets preempted, we should expect that Lazarus would get that flow run restarted, right? Seems like that's a primary use case for Lazarus.
If that's correct, sounds like we should expect to find something other than just preemption leading to what we're seeing
n
nicholas
04/20/2021, 4:35 PM
That sounds correct to me @Cab Maddux - let us know what you find and we can look to confirm further 👍