Hey all! We are running Prefect/Apollo/Hasura/Prefect-Agent (v 0.14.16). on Kubernetes on EC2 in AWS We run our Prefect-Agents as Local, not as K8S Agents. We find that when a Pod dies and is recreated by K8S a new Agent is created, registered, and becomes visible in the UI; however, the Agent that was killed when the Pod died turns into a Zombie and never gets cleaned up until we manually hit the Clean button a couple of times. When the agent count gets very high the entire platform bogs down, and the UI has trouble rendering. I thought that Towel was a service to help with this kind of thing - is there something special we need to do to get this set up?

Towel doesn’t help clearing the UI. How high is very high? You could schedule a job to hit the API endpoint to clear the agents I guess?

I could do that. We use spot instances for our lower environments and we've seen it get up in to the 1000's. We can probably just set up some kind of automation to click the "Clean" button if there's no out-of-the-box automation

There is none yep