https://prefect.io logo
#prefect-community
Title
# prefect-community
k

Kayvan Shah

05/22/2022, 11:10 AM
I observed that the pods start and keep failing and restarting While no resources show up when we get pods for default namespace
d.py
a

Anna Geller

05/22/2022, 11:26 AM
hard to determine the root cause just from those logs - are you running everything in a single K8s cluster? What storage do you use? could be your run can't pull the image or flow code
restarting your agent process may be a simple step worth trying
k

Kayvan Shah

05/22/2022, 11:38 AM
Running on a single k8s cluster
Also tried deleting everyting and restarting the minikube cluster but that didn't work as well
a

Anna Geller

05/22/2022, 11:40 AM
what did you do so far?
this is for the OSS version - single deployment with all components https://orion-docs.prefect.io/tutorials/kubernetes-flow-runner/
k

Kayvan Shah

05/22/2022, 11:40 AM
Following the same doc
can this happen when docker or minikube falls short of memory
a

Anna Geller

05/22/2022, 11:43 AM
What storage do you use? could be your run can't pull the image or flow code
yup for sure it can, good call
k

Kayvan Shah

05/22/2022, 11:43 AM
File Storage
a

Anna Geller

05/22/2022, 11:44 AM
file storage won't work with K8s afaik
you need remote storage such as S3, GCS, ABS
K8s cluster can't access your local files and mounting is hard/too much complexity when compared to e.g. just S3
k

Kayvan Shah

05/22/2022, 11:45 AM
will try with S3 Hope that works
I am trying to deploy on an on-premise server earlier when I had tried local storage creation of deployments failed but were created with file storage so I thought that things might work
a

Anna Geller

05/22/2022, 11:48 AM
gotcha, makes sense
k

Kayvan Shah

05/22/2022, 11:51 AM
Copy code
~/projects/prefect-demo$ minikube start
šŸ˜„  minikube v1.25.2 on Ubuntu 18.04 (kvm/amd64)
āœØ  Automatically selected the docker driver
šŸ‘  Starting control plane node minikube in cluster minikube
šŸšœ  Pulling base image ...
šŸ”„  Creating docker container (CPUs=2, Memory=3944MB) ...
šŸ³  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ā–Ŗ kubelet.housekeeping-interval=5m
    ā–Ŗ Generating certificates and keys ...
    ā–Ŗ Booting up control plane ...
    ā–Ŗ Configuring RBAC rules ...
šŸ”Ž  Verifying Kubernetes components...
    ā–Ŗ Using image <http://gcr.io/k8s-minikube/storage-provisioner:v5|gcr.io/k8s-minikube/storage-provisioner:v5>
šŸŒŸ  Enabled addons: storage-provisioner, default-storageclass
šŸ„  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
I think this much memory can suffice
Yet the pod with agent is not able to start and fails again & again
a

Anna Geller

05/22/2022, 12:02 PM
I see you are running it on Ubuntu 18.04 - do you have a chance to use Ubuntu 20.04 instead?
k

Kayvan Shah

05/22/2022, 12:07 PM
Have got ubuntu 20.04 lts can try running with wsl2
a

Anna Geller

05/22/2022, 12:13 PM
What's your current setup - is your end goal running Prefect on Windows? if so, we are working on this
k

Kayvan Shah

05/22/2022, 12:22 PM
Deploying prefect flows and workloads on AWS EKS. Currently working on POC on a company's on-premise Ubuntu 18.04 LTS server
a

Anna Geller

05/22/2022, 12:35 PM
in that case doing PoC on a small single node K8s cluster on AWS EKS may be easier, but definitely your choice how you decide to approach it, there's no right or wrong here - the link I shared is for AWS EKS
k

Kayvan Shah

05/22/2022, 2:48 PM
Even if I am running Docker and minikube on ubuntu 18.04 it shouldn't make much difference as prefect2b5 image may be built upon latest linux stable version?
Though the minikube deployment did not run as expected I was able to run most of the things locally
a

Anna Geller

05/22/2022, 6:41 PM
the consultant answer: it depends šŸ™‚
e.g. you may need to install Prefect to create deployments from this server - so you would need to install your orchestration layer e.g. on minikube and then the environment from which you create deployments would also need a 2.0 installation - if you install it on the same VM, then it's easier if it'S Ubuntu 20.04 than 18.04
āœ… 1
8 Views