Hello Having an issue with Kubernetes cluster autoscaler for

Question

Hello Having an issue with Kubernetes cluster autoscaler for long running gt 21 min flows similar to <https github com PrefectHQ prefect issues 3058|issue 3058> I noticed the issue has been marked closed What was the solution

Jim Crist-Harif · Answer

Hi Johnny the issue here is that the autoscaler will sometimes kill active jobs while prefect currently doesn t always like to have its jobs killed mid run The fix for you would be to modify the job template running on your agent to add the ``` <http cluster autoscaler kubernetes io safe to evict|cluster autoscaler kubernetes io safe to evict> false ``` label to the job template in `job spec template metadata labels` This will prevent the autoscaler from evicting active jobs

Jim Crist-Harif · Answer

To update the agent template you can copy the default template add the labels and whatever else you want save it in the same environment that the agent is running in then point the agent to it by setting the `YAML TEMPLATE` environment variable to where the template is located I recognize that this is a bit complicated we re currently working to simplify deployment configuration to make customizing deployments a lot simpler see

Johnny · Answer

thank you very very helpful info I ve been stuck on this for 2 days slightly smiling face

Jim Crist-Harif · Answer

Feel free to reach out sooner next time we re always happy to help Hope the above tips work for you

Johnny · Answer

will do