https://prefect.io
Join Slack
Menu
Home
Archive
Hey, i’m running a prefect agent on EKS / Fargate. It’s been working great for a while now, spinnin...
# ask-community
a
Hey, i’m running a prefect agent on EKS / Fargate. It’s been working great for a while now, spinning up a new pod, running and deleting it. I think for some reason something got changed on the AWS account and the spinning up of pods failed. However, it kept trying to retry and spin up a new pod, and the old pod didn’t get cleaned up. Now whenever I manually try to delete the pod (kubectl delete pods) it just recreates instantly. So now i have like 6 pods that are stuck in Pending status that I can’t delete. Any thoughts on how to clean everything up?
k
Maybe try 
kubectl delete jobs
instead of 
kubectl delete pods
?
e
This is a Kubernetes question. I can help
👍 2
🔥 1
So the question is: why the control plane of my Kubernetes cluster is not removing my resource ? Hard to say, However, you can force the delete of a resource using -f —grace-period=0 or something like that
Oh sorry I misread your question. The thing is that POD are a low level resource not meant to be created directly. I guess your pod were created by an higher level resource : a replica set, a deployment or a statefulset
What's happening is that the resource manager for the higher level resource is monitoring the pods and it is recreating them as you kill them. This is one of the critical and most important feature of Kubernetes. Can you list higher level resources ?
a
How would i list the higher level resources
I set everything up through the fargate tutorial
e
I have a bad news @Adi Gandra 🙂
Kubernetes is a powerful but not trivial solution. I would suggest you to do a tutorial using a local installation, such as Minikube, to understand these concepts
I am someone who is evaluating Prefect like you are, and I have been working with Kubernetes for three years so most likely I can help you solve this problem, but if you want to run workloads on Kubernetes, you need to understand it. Whether you use it with Fargate, Google Cloud, local installation
Copy code
kubectl ls deployment
a
Got it, that makes sense! We’ve had it running great for a couple months, but just running into some weird problems occassionally. I might just remake the fargate cluster
e
Wait 😄
I think that's right, you need to have reproducible environments. The right way to deploy a fargate cluster is using infrastructure as code, pick cloudformation (which I hate) or Terraform
👍 1
but unless you are using Prefect as a toy, you need to understand the infrastructure
Can you try to list all resources on your fargate cluster and show them here?
a
I’m really not sure what happened, but I only have one pod listed - the prefect agent. And when i do describe pod i get this error “Your AWS account has reached the limit on the number of Fargate pods it can run concurrently”
i deleted my old cluster, and setup a new one
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by