Hey, i’m running a prefect agent on EKS / Fargate....
# prefect-communitya
Adi Gandra
05/05/2022, 2:24 PMHey, i’m running a prefect agent on EKS / Fargate. It’s been working great for a while now, spinning up a new pod, running and deleting it. I think for some reason something got changed on the AWS account and the spinning up of pods failed. However, it kept trying to retry and spin up a new pod, and the old pod didn’t get cleaned up. Now whenever I manually try to delete the pod (kubectl delete pods) it just recreates instantly. So now i have like 6 pods that are stuck in Pending status that I can’t delete. Any thoughts on how to clean everything up?
k
Kevin Kho
05/05/2022, 2:29 PM
Maybe try
kubectl delete jobskubectl delete podse
Edmondo Porcu
05/05/2022, 2:52 PMThis is a Kubernetes question. I can help
👍 2
🔥 1
Edmondo Porcu
05/05/2022, 2:54 PMSo the question is: why the control plane of my Kubernetes cluster is not removing my resource ? Hard to say, However, you can force the delete of a resource using -f —grace-period=0 or something like that
Edmondo Porcu
05/05/2022, 2:55 PMOh sorry I misread your question. The thing is that POD are a low level resource not meant to be created directly. I guess your pod were created by an higher level resource : a replica set, a deployment or a statefulset
Edmondo Porcu
05/05/2022, 2:56 PMWhat's happening is that the resource manager for the higher level resource is monitoring the pods and it is recreating them as you kill them. This is one of the critical and most important feature of Kubernetes. Can you list higher level resources ?
a
Adi Gandra
05/05/2022, 3:22 PMHow would i list the higher level resources
Adi Gandra
05/05/2022, 3:22 PMI set everything up through the fargate tutorial
e
Edmondo Porcu
05/05/2022, 3:27 PMI have a bad news @Adi Gandra 🙂
Edmondo Porcu
05/05/2022, 3:28 PMKubernetes is a powerful but not trivial solution. I would suggest you to do a tutorial using a local installation, such as Minikube, to understand these concepts
Edmondo Porcu
05/05/2022, 3:29 PMI am someone who is evaluating Prefect like you are, and I have been working with Kubernetes for three years so most likely I can help you solve this problem, but if you want to run workloads on Kubernetes, you need to understand it. Whether you use it with Fargate, Google Cloud, local installation
Edmondo Porcu
05/05/2022, 3:30 PM Copy code
kubectl ls deploymenta
Adi Gandra
05/05/2022, 3:34 PMGot it, that makes sense! We’ve had it running great for a couple months, but just running into some weird problems occassionally. I might just remake the fargate cluster
e
Edmondo Porcu
05/05/2022, 3:36 PMWait 😄
Edmondo Porcu
05/05/2022, 3:36 PMI think that's right, you need to have reproducible environments. The right way to deploy a fargate cluster is using infrastructure as code, pick cloudformation (which I hate) or Terraform
👍 1
Edmondo Porcu
05/05/2022, 3:37 PMbut unless you are using Prefect as a toy, you need to understand the infrastructure
Edmondo Porcu
05/05/2022, 3:37 PMCan you try to list all resources on your fargate cluster and show them here?
a
Adi Gandra
05/06/2022, 3:54 AMI’m really not sure what happened, but I only have one pod listed - the prefect agent. And when i do describe pod i get this error “Your AWS account has reached the limit on the number of Fargate pods it can run concurrently”
Adi Gandra
05/06/2022, 3:54 AMi deleted my old cluster, and setup a new one
9 Views