Hi there, we are running the prefect agents in kubernetes as per https://docs.prefect.io/orchestration/agents/kubernetes.html#running-in-cluster. We notice if we don’t specify the agent name with

--name

, it will assume the default name

Kubernetes

. This is okay when you only run one replica, but when you increase the replica number to 2, both will take the default name and in the Prefect UI, the two agents are treated as the same agent. So what is the recommended way to deploy multiple agents for HA purposes in Kubernetes?

Ah so the reason they appear the same in the UI is so that if the agent disconnects and reconnects, we can treat it as the same and have the flow run history attached to it. But really, both agents can be submitted to so it does provide the redundancy.

But from the prefect automation’s perspective they are one agent? And if one of the two goes down, how does prefect detect it and alert on it? Clearly, we don’t want to wait till both go down to be alerted.

Ah in that case you would need to split them as different name but same labels. Yes they are treated as the same for automations

which means we need to repeat the same manifest again just for a different name? mehhhh

Yeah I don’t see another way because the Automation is just us detecting something the agent hasn’t pinged us, but we can’t really agents current with the info they provide. The other side of this would be that every agent stop and start is a new agent in the UI

is there a plan allowing agents to expose metrics to be scraped by telemetry systems such as Prometheus?

Not plan for Prefect 1 but potential roadmap item for Prefect 2

k got it - thanks Kevin!