Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Adi Gandra
05/05/2022, 2:24 PMHey, i’m running a prefect agent on EKS / Fargate. It’s been working great for a while now, spinning up a new pod, running and deleting it. I think for some reason something got changed on the AWS account and the spinning up of pods failed. However, it kept trying to retry and spin up a new pod, and the old pod didn’t get cleaned up. Now whenever I manually try to delete the pod (kubectl delete pods) it just recreates instantly. So now i have like 6 pods that are stuck in Pending status that I can’t delete. Any thoughts on how to clean everything up?
k
Kevin Kho
05/05/2022, 2:29 PMMaybe try
kubectl delete jobskubectl delete podse
Edmondo Porcu
05/05/2022, 2:52 PMThis is a Kubernetes question. I can help
👍 2
🔥 1
So the question is: why the control plane of my Kubernetes cluster is not removing my resource ? Hard to say, However, you can force the delete of a resource using -f —grace-period=0 or something like that
Oh sorry I misread your question. The thing is that POD are a low level resource not meant to be created directly. I guess your pod were created by an higher level resource : a replica set, a deployment or a statefulset
What's happening is that the resource manager for the higher level resource is monitoring the pods and it is recreating them as you kill them. This is one of the critical and most important feature of Kubernetes. Can you list higher level resources ?
a
Adi Gandra
05/05/2022, 3:22 PMHow would i list the higher level resources
I set everything up through the fargate tutorial
e
Edmondo Porcu
05/05/2022, 3:27 PMI have a bad news @Adi Gandra 🙂
Kubernetes is a powerful but not trivial solution. I would suggest you to do a tutorial using a local installation, such as Minikube, to understand these concepts
I am someone who is evaluating Prefect like you are, and I have been working with Kubernetes for three years so most likely I can help you solve this problem, but if you want to run workloads on Kubernetes, you need to understand it. Whether you use it with Fargate, Google Cloud, local installation
kubectl ls deploymenta
Adi Gandra
05/05/2022, 3:34 PMGot it, that makes sense! We’ve had it running great for a couple months, but just running into some weird problems occassionally. I might just remake the fargate cluster
e
Edmondo Porcu
05/05/2022, 3:36 PMWait 😄
I think that's right, you need to have reproducible environments. The right way to deploy a fargate cluster is using infrastructure as code, pick cloudformation (which I hate) or Terraform
👍 1
but unless you are using Prefect as a toy, you need to understand the infrastructure
Can you try to list all resources on your fargate cluster and show them here?
a
Adi Gandra
05/06/2022, 3:54 AMI’m really not sure what happened, but I only have one pod listed - the prefect agent. And when i do describe pod i get this error “Your AWS account has reached the limit on the number of Fargate pods it can run concurrently”
i deleted my old cluster, and setup a new one