Igor Morgunov

10/04/2022, 3:54 PM
Hi All. I have an issue with failed flows not deleting their pods in kubernetes. If I run a flow, and then start another run of the same flow, the second (or any subsequent) run will fail with kubernetes throwing a 409, but the pod it was running on doesn’t get deleted and remains in running state - any suggestions how to get around this?

Mason Menges

10/04/2022, 6:38 PM
Hey @Igor Morgunov The short answer to this is that we're working to improve the workflow around these kinds of events. The longer answer is that when the underlying infrastructure running a flow disappears there's no longer anything communicating with the prefect cloud API to let it know the status of the flow run, i.e. there's no indication to Cloud that the pod running the flow was actually terminated so we don't have any visibility into what's occurring. This pose a few challenges in terms of how we could address this and it's something we're having active discussions around to provide the best functionality/visibility for the time being the simplest way to address these flow runs is to delete them. Ignore this I had misread the initial question 😅


10/04/2022, 6:38 PM
How is your flow failing? The process should exit when a flow fails.
(And consequently, the pod should exit)