https://prefect.io logo
#prefect-community
Title
# prefect-community
a

Adi Gandra

05/05/2022, 2:24 PM
Hey, i’m running a prefect agent on EKS / Fargate. It’s been working great for a while now, spinning up a new pod, running and deleting it. I think for some reason something got changed on the AWS account and the spinning up of pods failed. However, it kept trying to retry and spin up a new pod, and the old pod didn’t get cleaned up. Now whenever I manually try to delete the pod (kubectl delete pods) it just recreates instantly. So now i have like 6 pods that are stuck in Pending status that I can’t delete. Any thoughts on how to clean everything up?
k

Kevin Kho

05/05/2022, 2:29 PM
Maybe try
kubectl delete jobs
instead of
kubectl delete pods
?
e

Edmondo Porcu

05/05/2022, 2:52 PM
This is a Kubernetes question. I can help
👍 2
🔥 1
So the question is: why the control plane of my Kubernetes cluster is not removing my resource ? Hard to say, However, you can force the delete of a resource using -f —grace-period=0 or something like that
Oh sorry I misread your question. The thing is that POD are a low level resource not meant to be created directly. I guess your pod were created by an higher level resource : a replica set, a deployment or a statefulset
What's happening is that the resource manager for the higher level resource is monitoring the pods and it is recreating them as you kill them. This is one of the critical and most important feature of Kubernetes. Can you list higher level resources ?
a

Adi Gandra

05/05/2022, 3:22 PM
How would i list the higher level resources
I set everything up through the fargate tutorial
e

Edmondo Porcu

05/05/2022, 3:27 PM
I have a bad news @Adi Gandra 🙂
Kubernetes is a powerful but not trivial solution. I would suggest you to do a tutorial using a local installation, such as Minikube, to understand these concepts
I am someone who is evaluating Prefect like you are, and I have been working with Kubernetes for three years so most likely I can help you solve this problem, but if you want to run workloads on Kubernetes, you need to understand it. Whether you use it with Fargate, Google Cloud, local installation
Copy code
kubectl ls deployment
a

Adi Gandra

05/05/2022, 3:34 PM
Got it, that makes sense! We’ve had it running great for a couple months, but just running into some weird problems occassionally. I might just remake the fargate cluster
e

Edmondo Porcu

05/05/2022, 3:36 PM
Wait 😄
I think that's right, you need to have reproducible environments. The right way to deploy a fargate cluster is using infrastructure as code, pick cloudformation (which I hate) or Terraform
👍 1
but unless you are using Prefect as a toy, you need to understand the infrastructure
Can you try to list all resources on your fargate cluster and show them here?
a

Adi Gandra

05/06/2022, 3:54 AM
I’m really not sure what happened, but I only have one pod listed - the prefect agent. And when i do describe pod i get this error “Your AWS account has reached the limit on the number of Fargate pods it can run concurrently”
i deleted my old cluster, and setup a new one
9 Views