https://prefect.io
Join Slack
Hi folks, I am facing an issue unable to remove an agent in prefect 1 (cloud) - I tried doing so via...
# ask-community
m
Hi folks, I am facing an issue unable to remove an agent in prefect 1 (cloud) - I tried doing so via graphql and the UI to no avail
1️⃣ 1
m
Hey @Marwan Sarieddine Could you elaborate a bit on what you're seeing, is the infrastructure that was running the agent actually shutdown? are you seeing any error when trying to remove it from the UI?
m
@Mason Menges thank you for following up on this So this is a kubernetes agent that I took down from the infrastructure that was running it It has been lingering for more than a week on the prefect cloud UI (under https://cloud.prefect.io/tenant/agent as an unhealthy agent and everytime I try to remove it I get this error shown in the screenshot
Copy code
We had a problem removing your Agent. Please try again.
When I attempt to run this graphql mutation
Copy code
mutation{
  delete_agent(input:{
    agent_id:"e2b38599-a3e5-4b9d-b1c5-a3ee6076c9d8"
  }){
    success,
    error
  }
}
I get this error
Copy code
{
  "errors": [
    {
      "path": [
        "delete_agent"
      ],
      "message": "Operation timed out",
      "extensions": {
        "code": "API_ERROR"
      }
    }
  ],
  "data": {
    "delete_agent": null
  }
}
The unfortunate part is I believe this agent is still being taken into consideration when trying to submit runs. It shares the same labels with another agent but its labels are only a subset of the other agent, this results in our flows getting stuck in a scheduled state
m
Thanks for the additional details, I'll look around a bit and see what I can dig up, we're doing some maintenance this weekend on our backend that might help with this as well but I'll see what I can find in the mean time.
m
Thank you
m
Hey @Marwan Sarieddine digging around it's possible that there might still be flows runs that are referencing this agent, this might happen when the agent is responsible for submitting a particularly large number of flow runs, you might see if you can filter for flow runs submitted by this agent and see if any are stuck in a non-final state and if so see if you can manually update their state.
m
Thanks for the suggestion - so if I stop all stuck runs then this might unblock the agent ?
I will confirm if this is the case and if it resolves things
m
In theory yes 😅
🤞 1
m
@Mason Menges this past weekend I stopped all running flow runs and attempted a delete on the agent, it did not work - still getting the same error please see the attached screenshots
Attempting the delete using a graphql mutation returns an API timeout error
m
Hmm, Have you tried turning off the schedules for the flows that are submitting work to this agent? This should remove any future scheduled runs as well as any submittable runs from the agent and should hopefully allow you to delete it.
the build up of scheduled runs on it might also be preventing this from getting removed
m
Oh I did not turn off the schedules, ok I will attempt to do so this weekend
thanks for the quick follow up
So I removed all scheduled flow runs - still getting the same timeout error and I am not able to delete the agent
@Mason Menges
interesting is that your suggestion partially resolved the issue - i.e. there were three agents that were stuck and I am now down to only one…
22 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by