hello. I’m running prefect agents on k8s with 3 replicas and I’m seeing some runs being submitted to more than 1 pod.
After pod A completes the submission, pod B picks up the same run and crashes because it can’t submit it again (our infrastructure will only allow a single submission for a given flow run ID). The flow submitted via pod A still runs but there’s a brief period when the run is marked as Crashed in the UI.
Is anyone else facing a similar issue?
Timeline and screenshots:
• 211227 - run submitted to pod A (screenshot 1)
• 211236 - submission completed by pod A (screenshot 2)
• 211241 - run submitted to pod B (screenshot 3)
• 211245 - crash in pod B (screenshot 4)
• 214041 - submission via pod A finishes (screenshot 5)