We ran into a similar where what was happening is the request to delete the agent was timing out
We resolved this by temporarily disabling the schedules on all of the flows based on a suggestion from a member of the prefect team i.e remove all future scheduled flow runs first and then attempt the agent delete, this way the query can run within the allotted timeout period