Hello all Since late last week we have noticed Prefect flows

Question

Hello all Since late last week we have noticed Prefect flows running in k8s showing as Crashed in the Prefect Cloud UI Using k9s to inspect the pods has revealed that the pods were in fact still running Sometimes the flows have sprung suddenly from Crashed to Running and completed successfully One that Crashed yesterday apparently still has the associated pod running What might account for this What do you recommend we do Thanks in advance

Nate · Answer

hi < Jeff Charbeneau> do you have the pod logs from a run where pods were inappropriately marked as crashed if you had the worker logs from this time that would also be helpful also I ve assumed you re using the k8s worker here and not the agent is that true

Jeff Charbeneau · Answer

< Nate> Thanks for being so responsive Let me see if I can recreate this behavior and collect some pod logs I ll also confirm that we re using the Prefect k8s agent and not the k8s worker as I expect

Jeff Charbeneau · Answer

< Nate> In k9s I see a `prefect 2 agent ` pod leading me to believe we re using the Prefect k8s agent and not the k8s worker Below are some pod logs from a recent flow run that had the described behavior In the Prefect UI the flow changed from Crashed to Running around the time that this log message was printed ```19 21 05 042 | INFO | Flow run phi652 alterf Downloading flow code from storage at None``` Thanks in advance for your thoughts

Jeff Charbeneau · Answer

< Nate> How much of a chance have you had to look at the logs

Nate · Answer

hey < Jeff Charbeneau> afaict it seems like the failure you shared occurs during within the c4 job manager ```c4 job manager c4 error C4Error Job be7bajuioxg6gvfnhcufc6 failed detail= output= For logs see ``` can you explain what this is is there some way it would revive a process that prefect has lost touch with after a Crash of sorts

Jeff Charbeneau · Answer

Thanks for taking a look < Nate> You re right there is a failure in the flow However that failure occurs after the flow changes from Crashed where it starts to Running Furthermore we ve got retries on that task and are able to recover from the failure Again the issue is that the flow starts in a Crashed state the pod is still running and then the flow state changes to Running

Nate · Answer

hmm sorry if im just missing it but where are you seeing the transition to Crashed in the logs if not the logs is it just in the UI

Jeff Charbeneau · Answer

Great question Nate Crashed appears only in the Prefect UI not in the logs