https://prefect.io logo
Join Slack
Powered by
I observed that the pods start and keep failing an...
# ask-community
k
I observed that the pods start and keep failing and restarting While no resources show up when we get pods for default namespace
error.txt.py
d.py
d.py
a
hard to determine the root cause just from those logs - are you running everything in a single K8s cluster? What storage do you use? could be your run can't pull the image or flow code
restarting your agent process may be a simple step worth trying
k
Running on a single k8s cluster
Also tried deleting everyting and restarting the minikube cluster but that didn't work as well
a
what did you do so far?
this is for the OSS version - single deployment with all components https://orion-docs.prefect.io/tutorials/kubernetes-flow-runner/
k
Following the same doc
can this happen when docker or minikube falls short of memory
a
What storage do you use? could be your run can't pull the image or flow code
yup for sure it can, good call
k
File Storage
a
file storage won't work with K8s afaik
you need remote storage such as S3, GCS, ABS
K8s cluster can't access your local files and mounting is hard/too much complexity when compared to e.g. just S3
k
will try with S3 Hope that works
I am trying to deploy on an on-premise server earlier when I had tried local storage creation of deployments failed but were created with file storage so I thought that things might work
a
gotcha, makes sense
k
Copy code
~/projects/prefect-demo$ minikube start
šŸ˜„  minikube v1.25.2 on Ubuntu 18.04 (kvm/amd64)
āœØ  Automatically selected the docker driver
šŸ‘  Starting control plane node minikube in cluster minikube
šŸšœ  Pulling base image ...
šŸ”„  Creating docker container (CPUs=2, Memory=3944MB) ...
šŸ³  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ā–Ŗ kubelet.housekeeping-interval=5m
    ā–Ŗ Generating certificates and keys ...
    ā–Ŗ Booting up control plane ...
    ā–Ŗ Configuring RBAC rules ...
šŸ”Ž  Verifying Kubernetes components...
    ā–Ŗ Using image <http://gcr.io/k8s-minikube/storage-provisioner:v5|gcr.io/k8s-minikube/storage-provisioner:v5>
šŸŒŸ  Enabled addons: storage-provisioner, default-storageclass
šŸ„  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
I think this much memory can suffice
Yet the pod with agent is not able to start and fails again & again
a
I see you are running it on Ubuntu 18.04 - do you have a chance to use Ubuntu 20.04 instead?
k
Have got ubuntu 20.04 lts can try running with wsl2
a
What's your current setup - is your end goal running Prefect on Windows? if so, we are working on this
k
Deploying prefect flows and workloads on AWS EKS. Currently working on POC on a company's on-premise Ubuntu 18.04 LTS server
a
in that case doing PoC on a small single node K8s cluster on AWS EKS may be easier, but definitely your choice how you decide to approach it, there's no right or wrong here - the link I shared is for AWS EKS
k
Even if I am running Docker and minikube on ubuntu 18.04 it shouldn't make much difference as prefect2b5 image may be built upon latest linux stable version?
Though the minikube deployment did not run as expected I was able to run most of the things locally
a
the consultant answer: it depends šŸ™‚
e.g. you may need to install Prefect to create deployments from this server - so you would need to install your orchestration layer e.g. on minikube and then the environment from which you create deployments would also need a 2.0 installation - if you install it on the same VM, then it's easier if it'S Ubuntu 20.04 than 18.04
āœ… 1
64 Views
Open in Slack
PreviousNext
Powered by