hi devs!, anyone else facing these issues ?
1. is there a way to get logs of crashed prefect flows running as kubernetes jobs on EKS?
2. flows on prefect go to crashed state after duration of 6 hours. but they still keep running as kubernetes job in background . any reason for this incorrect status
3. some of the flows show running status on the ui but the kubernetes job for them is already ended. these running flows choke the queue limit and the task concurrency limit shared by similar flows
đź‘€ 1
c
Christopher Boyd
08/08/2023, 2:04 PM
Hi Deepanshu , for Crashed flows I would look to cloud watch - if it’s an infrastructure exception and not flow code, prefect can’t capture the even since it’s a process inside the container .
For 2- I think we would need to see logs and something else - perhaps the job crashed on prefect side but was restarted in k8s side as a retried job ? There is a lot of missing detail to evaluate that one .
3 - do you have any sort of hooks enabled ? Are these sun flows at all ? What version of prefect ? We have seen reports of this happening but haven’t been able to reproduce so any observations and details here are useful
d
Deepanshu Aggarwal
08/08/2023, 2:07 PM
for 2. we have not enabled restarts. so it just runs normally as kubernetes job but on the ui it shows as crashed
for 3. not clear what are sun flows. the version of prefect we are using is 2.7.8 (we are looking to upgrade it soon)
im not sure how to reproduce this , because it happens at random.
c
Christopher Boyd
08/08/2023, 2:11 PM
That was meant to be sub flows , autocorrect took over
Christopher Boyd
08/08/2023, 2:11 PM
And yes, there have been a vast number of fixes and introduction of workers
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.