<@ULVA73B9P> I try to self host in K3s cluster Pre...
# ask-marvin
l
@Marvin I try to self host in K3s cluster Prefect (server AND workers). When running prefect script, the prefect-server completely crash (make workers crash too because they can't make request to api). I was wondering what are the minimum requirement for prefect-server ?
m
thought for 206.9 seconds
Thanks for the details — a crashing API will definitely cascade to workers since they can’t poll the queue. Assuming you’re on Prefect 3.x, here’s a practical baseline for K3s plus some quick diagnostics to confirm if this is just resource starvation (very common on k3s) vs. misconfig. Quick answer: minimum practical resources - Prefect Server (API + services): - Start with: requests = 500m CPU / 512Mi memory, limits = 1 CPU / 1–2Gi memory - If you’re very constrained, you can try: requests = 200–300m CPU / 256–384Mi memory, limits = 500m CPU / 512–768Mi memory (but expect OOMKills under load) - Prefect Worker (control-plane pod, not your job pods): - requests = 100–200m CPU / 256–512Mi memory, limits = 1 CPU / 1Gi memory - Database: - Use Postgres for anything beyond a tiny single-server test - Small K3s Postgres: requests ~100m / 256Mi; limits ~500m / 512Mi; 10Gi+ storage If you’re using the Prefect Helm charts, these values mirror/track the chart defaults and work well for small clusters: - Server resources (values.yaml)
Copy code
server:
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "1"
      memory: 1Gi

backgroundServices:
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "1"
      memory: 1Gi
- Worker resources (values.yaml)
Copy code
worker:
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: "1"
      memory: 1Gi
Common K3s gotchas that cause server crashes - OOMKilled: the most frequent cause. Check
kubectl describe pod
for OOMKilled and bump memory limits/requests as above. - CPU starvation/throttling: K3s nodes with 1 vCPU often throttle; set a proper CPU request so the scheduler reserves it. - SQLite with multiple replicas: use Postgres if you scale server replicas beyond 1. - DB connection pool too large for tiny Postgres: reduce pool if you’re very resource constrained:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=3
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=5
- Ingress / service misconfig: workers must reach
PREFECT_API_URL
(test with curl from a worker pod). Fast debug checklist 1) Describe the crashing pod
Copy code
kubectl describe pod <prefect-server-pod>
kubectl logs <prefect-server-pod> -c server
kubectl logs <prefect-server-pod> -c services
- Look for OOMKilled, CrashLoopBackOff, or db connection errors. 2) Verify DB is reachable and healthy (pods up, PVC size, logs show success). 3) Confirm worker API URL/Key
Copy code
echo $PREFECT_API_URL
echo $PREFECT_API_KEY
4) From a worker pod, test reachability
Copy code
apk add curl || apt-get update && apt-get install -y curl
curl -sS <your-api-url>/api/health
If you want an even leaner K3s profile to try first (single-node dev/test)
Copy code
server:
  resources:
    requests: { cpu: 200m, memory: 256Mi }
    limits:   { cpu: 500m, memory: 512Mi }
backgroundServices:
  resources:
    requests: { cpu: 200m, memory: 256Mi }
    limits:   { cpu: 500m, memory: 512Mi }
But if you see any OOMKilled, jump to 512Mi requests / 1Gi limits immediately. Cluster sizing guidance (smallest workable) - Single-node K3s dev box: at least 2 vCPU and 2–4 GiB free for Prefect + Postgres - HA or heavier use: 8–12 GiB RAM cluster-wide and 4–6 vCPU Useful references - Helm charts (server + worker): prefect-helm - Server chart values: prefect-server/values.yaml - Worker chart values: prefect-worker/values.yaml If you can share: - How you deployed (Helm chart version/values or manifests) - k describe/k logs from the crashing server pod - Node resources (CPU/RAM), and whether you’re using Postgres or SQLite …I can pinpoint exactly why the server is crashing and give you a tuned values.yaml for your cluster.
l
@Marvin So in my argocd prefect yaml :
Copy code
apiVersion: <http://argoproj.io/v1alpha1|argoproj.io/v1alpha1>
kind: Application
metadata:
  name: prefect
  namespace: argocd
spec:
  project: default

  sources:
    - repoURL: <https://prefecthq.github.io/prefect-helm>
      chart: prefect-server
      targetRevision: 2025.11.21180339
      helm:
        valuesObject:
          server:
            replicaCount: 1
            
            basicAuth:
              enabled: true
              existingSecret: "prefect-auth-secret"
            
            uiConfig:
              prefectUiApiUrl: "<https://prefect.recomend.app/api>"
            
            env:
              - name: PREFECT_SERVER_API_CSRF_PROTECTION_ENABLED
                value: "true"

          postgresql:
            enabled: true
            primary:
              persistence:
                enabled: true
                storageClass: hcloud-volumes
                size: 10Gi
                annotations:
                  <http://resize.topolvm.io/storage_limit|resize.topolvm.io/storage_limit>: "50Gi"
                  <http://resize.topolvm.io/threshold|resize.topolvm.io/threshold>: "20%"
                  <http://resize.topolvm.io/increase|resize.topolvm.io/increase>: "25%"
                  <http://backup.velero.io/backup-volumes|backup.velero.io/backup-volumes>: "data"

    - repoURL: <https://prefecthq.github.io/prefect-helm>
      chart: prefect-worker
      targetRevision: 2025.11.21180339
      helm:
        valuesObject:
          worker:
            apiConfig: selfHostedServer
            config:
              workPool: kube
              baseJobTemplate:
                existingConfigMapName: prefect-base-job-template
              
            selfHostedServerApiConfig:
              apiUrl: "<http://prefect-server.prefect.svc.cluster.local:4200/api>"

              basicAuth:
                enabled: true
                existingSecret: "prefect-auth-secret"

    - repoURL: <https://github.com/recomendapp/infra.git>
      targetRevision: main
      path: apps/services/prefect
      directory:
        recurse: true

  destination:
    server: <https://kubernetes.default.svc>
    namespace: prefect

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
How can I set minimum ressources to the server ?
m
thought for 97.5 seconds
Great question — you can set resource requests/limits via the Helm values for the prefect-server chart. In your ArgoCD Application, add a
server.resources
block under the prefect-server source. Example (safe minimum for k3s; bump if you see OOMKilled):
Copy code
sources:
  - repoURL: <https://prefecthq.github.io/prefect-helm>
    chart: prefect-server
    targetRevision: 2025.11.21180339
    helm:
      valuesObject:
        server:
          replicaCount: 1

          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "1"
              memory: 1Gi

          basicAuth:
            enabled: true
            existingSecret: "prefect-auth-secret"

          uiConfig:
            prefectUiApiUrl: "<https://prefect.recomend.app/api>"

          env:
            - name: PREFECT_SERVER_API_CSRF_PROTECTION_ENABLED
              value: "true"
If you’re extremely constrained and just testing, you can try a leaner profile (expect to bump if pods get OOMKilled):
Copy code
server:
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi
If you enable background services as a separate deployment, also set their resources:
Copy code
backgroundServices:
  runAsSeparateDeployment: true
  resources:
    requests:
      cpu: 200–500m
      memory: 256–512Mi
    limits:
      cpu: 500m–1
      memory: 512Mi–1Gi
Optional (migration job):
Copy code
migrations:
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 256Mi
Links to chart values for reference: - Server chart values: prefect-server/values.yaml - Worker chart values: prefect-worker/values.yaml Tip: after syncing, if the server still crashes, run
kubectl describe pod
on the server pod to check for OOMKilled; if so, raise the memory limit to 1–2Gi.