I observed that the pods start and keep failing and restarti

Question

I observed that the pods start and keep failing and restarting While no resources show up when we get pods for default namespace

Kayvan Shah · Answer

d py

Anna Geller · Answer

hard to determine the root cause just from those logs are you running everything in a single K8s cluster What storage do you use could be your run can t pull the image or flow code

Anna Geller · Answer

restarting your agent process may be a simple step worth trying

Kayvan Shah · Answer

Running on a single k8s cluster

Kayvan Shah · Answer

Also tried deleting everyting and restarting the minikube cluster but that didn t work as well

Anna Geller · Answer

what did you do so far

Anna Geller · Answer

this is for the OSS version single deployment with all components <https orion docs prefect io tutorials kubernetes flow runner >

Anna Geller · Answer

and for Cloud 2 0 you can follow this one <https discourse prefect io t prefect 2 0 and prefect cloud 2 0 overview and work queues by laura lorenz 840>

Kayvan Shah · Answer

Following the same doc

Kayvan Shah · Answer

can this happen when docker or minikube falls short of memory

Anna Geller · Answer

What storage do you use could be your run can t pull the image or flow code

Anna Geller · Answer

yup for sure it can good call

Kayvan Shah · Answer

File Storage

Anna Geller · Answer

file storage won t work with K8s afaik

Anna Geller · Answer

you need remote storage such as S3 GCS ABS

Anna Geller · Answer

K8s cluster can t access your local files and mounting is hard too much complexity when compared to e g just S3

Kayvan Shah · Answer

will try with S3 Hope that works

Kayvan Shah · Answer

I am trying to deploy on an on premise server earlier when I had tried local storage creation of deployments failed but were created with file storage so I thought that things might work

Anna Geller · Answer

gotcha makes sense

Kayvan Shah · Answer

```~ projects prefect demo$ minikube start minikube v1 25 2 on Ubuntu 18 04 kvm amd64 Automatically selected the docker driver Starting control plane node minikube in cluster minikube Pulling base image Creating docker container CPUs=2 Memory=3944MB Preparing Kubernetes v1 23 3 on Docker 20 10 12 kubelet housekeeping interval=5m Generating certificates and keys Booting up control plane Configuring RBAC rules Verifying Kubernetes components Using image <http gcr io k8s minikube storage provisioner v5|gcr io k8s minikube storage provisioner v5> Enabled addons storage provisioner default storageclass Done kubectl is now configured to use minikube cluster and default namespace by default``` I think this much memory can suffice

Kayvan Shah · Answer

Yet the pod with agent is not able to start and fails again amp again

Anna Geller · Answer

I see you are running it on Ubuntu 18 04 do you have a chance to use Ubuntu 20 04 instead

Kayvan Shah · Answer

Have got ubuntu 20 04 lts can try running with wsl2

Anna Geller · Answer

What s your current setup is your end goal running Prefect on Windows if so we are working on this

Kayvan Shah · Answer

Deploying prefect flows and workloads on AWS EKS Currently working on POC on a company s on premise Ubuntu 18 04 LTS server

Anna Geller · Answer

in that case doing PoC on a small single node K8s cluster on AWS EKS may be easier but definitely your choice how you decide to approach it there s no right or wrong here the link I shared is for AWS EKS

Kayvan Shah · Answer

Even if I am running Docker and minikube on ubuntu 18 04 it shouldn t make much difference as prefect2b5 image may be built upon latest linux stable version

Kayvan Shah · Answer

Though the minikube deployment did not run as expected I was able to run most of the things locally

Anna Geller · Answer

the consultant answer it depends slightly smiling face

Anna Geller · Answer

e g you may need to install Prefect to create deployments from this server so you would need to install your orchestration layer e g on minikube and then the environment from which you create deployments would also need a 2 0 installation if you install it on the same VM then it s easier if it S Ubuntu 20 04 than 18 04