Hi folks, I am facing an issue unable to remove an...
# ask-communitym
Marwan Sarieddine
12/04/2022, 2:26 PMHi folks, I am facing an issue unable to remove an agent in prefect 1 (cloud) - I tried doing so via graphql and the UI to no avail
1️⃣ 1
m
Mason Menges
12/05/2022, 7:10 PMHey @Marwan Sarieddine Could you elaborate a bit on what you're seeing, is the infrastructure that was running the agent actually shutdown? are you seeing any error when trying to remove it from the UI?
m
Marwan Sarieddine
12/05/2022, 7:19 PM@Mason Menges thank you for following up on this
So this is a kubernetes agent that I took down from the infrastructure that was running it
It has been lingering for more than a week on the prefect cloud UI (under
https://cloud.prefect.io/tenant/agent as an unhealthy agent
and everytime I try to remove it I get this error shown in the screenshot
Copy code
We had a problem removing your Agent. Please try again.Marwan Sarieddine
12/05/2022, 7:20 PMWhen I attempt to run this graphql mutation
Copy code
mutation{
delete_agent(input:{
agent_id:"e2b38599-a3e5-4b9d-b1c5-a3ee6076c9d8"
}){
success,
error
}
} Copy code
{
"errors": [
{
"path": [
"delete_agent"
],
"message": "Operation timed out",
"extensions": {
"code": "API_ERROR"
}
}
],
"data": {
"delete_agent": null
}
}Marwan Sarieddine
12/05/2022, 7:21 PMThe unfortunate part is I believe this agent is still being taken into consideration when trying to submit runs. It shares the same labels with another agent but its labels are only a subset of the other agent, this results in our flows getting stuck in a scheduled state
m
Mason Menges
12/05/2022, 7:27 PMThanks for the additional details, I'll look around a bit and see what I can dig up, we're doing some maintenance this weekend on our backend that might help with this as well but I'll see what I can find in the mean time.
m
Marwan Sarieddine
12/05/2022, 7:27 PMThank you
m
Mason Menges
12/07/2022, 4:30 PMHey @Marwan Sarieddine digging around it's possible that there might still be flows runs that are referencing this agent, this might happen when the agent is responsible for submitting a particularly large number of flow runs, you might see if you can filter for flow runs submitted by this agent and see if any are stuck in a non-final state and if so see if you can manually update their state.
m
Marwan Sarieddine
12/07/2022, 4:31 PMThanks for the suggestion - so if I stop all stuck runs then this might unblock the agent ?
Marwan Sarieddine
12/07/2022, 4:31 PMI will confirm if this is the case and if it resolves things
m
Mason Menges
12/07/2022, 4:34 PMIn theory yes 😅
🤞 1
m
Marwan Sarieddine
12/21/2022, 9:51 PM@Mason Menges this past weekend I stopped all running flow runs and attempted a delete on the agent, it did not work - still getting the same error
please see the attached screenshots
Marwan Sarieddine
12/21/2022, 9:52 PMAttempting the delete using a graphql mutation returns an API timeout error
m
Mason Menges
12/21/2022, 9:57 PMHmm, Have you tried turning off the schedules for the flows that are submitting work to this agent? This should remove any future scheduled runs as well as any submittable runs from the agent and should hopefully allow you to delete it.
Mason Menges
12/21/2022, 9:57 PMthe build up of scheduled runs on it might also be preventing this from getting removed
m
Marwan Sarieddine
12/21/2022, 9:57 PMOh I did not turn off the schedules, ok I will attempt to do so this weekend
Marwan Sarieddine
12/21/2022, 9:57 PMthanks for the quick follow up
Marwan Sarieddine
12/27/2022, 2:02 PMSo I removed all scheduled flow runs - still getting the same timeout error and I am not able to delete the agent
Marwan Sarieddine
12/27/2022, 2:03 PM@Mason Menges
Marwan Sarieddine
12/27/2022, 2:24 PMinteresting is that your suggestion partially resolved the issue - i.e. there were three agents that were stuck and I am now down to only one…
23 Views