https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I try to setup Prefect in k3s : ```ap...
# ask-marvin
l
@Marvin I try to setup Prefect in k3s :
Copy code
apiVersion: <http://argoproj.io/v1alpha1|argoproj.io/v1alpha1>
kind: Application
metadata:
  name: prefect
  namespace: argocd
spec:
  project: default

  sources:
    - repoURL: <https://prefecthq.github.io/prefect-helm>
      chart: prefect-server
      targetRevision: 2025.11.21180339
      helm:
        valuesObject:
          server:
            replicaCount: 1

            resources:
              requests:
                cpu: 500m
                memory: 512Mi
              limits:
                cpu: "1"
                memory: 1Gi
            
            basicAuth:
              enabled: true
              existingSecret: "prefect-auth-secret"
            
            uiConfig:
              prefectUiApiUrl: "<https://prefect.recomend.app/api>"
            
            env:
              - name: PREFECT_SERVER_API_CSRF_PROTECTION_ENABLED
                value: "true"

          postgresql:
            enabled: true
            primary:
              podAnnotations:
                <http://backup.velero.io/backup-volumes|backup.velero.io/backup-volumes>: "data"
              persistence:
                enabled: true
                storageClass: hcloud-volumes
                size: 10Gi
                annotations:
                  <http://resize.topolvm.io/storage_limit|resize.topolvm.io/storage_limit>: "50Gi"
                  <http://resize.topolvm.io/threshold|resize.topolvm.io/threshold>: "20%"
                  <http://resize.topolvm.io/increase|resize.topolvm.io/increase>: "25%"

    - repoURL: <https://prefecthq.github.io/prefect-helm>
      chart: prefect-worker
      targetRevision: 2025.11.21180339
      helm:
        valuesObject:
          worker:
            apiConfig: selfHostedServer
            config:
              workPool: kube
              baseJobTemplate:
                existingConfigMapName: prefect-base-job-template
              
            selfHostedServerApiConfig:
              apiUrl: "<http://prefect-server.prefect.svc.cluster.local:4200/api>"

              basicAuth:
                enabled: true
                existingSecret: "prefect-auth-secret"

    - repoURL: <https://github.com/recomendapp/infra.git>
      targetRevision: main
      path: apps/services/prefect
      directory:
        recurse: true

  destination:
    server: <https://kubernetes.default.svc>
    namespace: prefect

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
ANd backup postgresql db of prefect with velero. But right now, the annotation dont work. How to make it work ?
m
✅ thought for 296.3 seconds
Short answer: add the Velero annotation to the PostgreSQL pod template and make sure the Velero node-agent is installed. In the Bitnami PostgreSQL subchart that Prefect uses, the volume you want is named 
data
, and the correct values path is 
postgresql.primary.podAnnotations
. A few things to check/fix: - No Slack-style link markup in YAML keys. The key must be exactly 
<http://backup.velero.io/backup-volumes|backup.velero.io/backup-volumes>
(not 
<http://...|...>
). Same for the TopoLVM keys. - Put the Velero annotation on the pod template (not the PVC). For the Prefect chart, that’s:
Copy code
postgresql:
    enabled: true
    primary:
      podAnnotations:
        <http://backup.velero.io/backup-volumes|backup.velero.io/backup-volumes>: "data"
      persistence:
        enabled: true
        storageClass: hcloud-volumes
        size: 10Gi
        # NOTE: These TopoLVM annotations won’t do anything on hcloud-volumes
        annotations:
          <http://resize.topolvm.io/storage_limit|resize.topolvm.io/storage_limit>: "50Gi"
          <http://resize.topolvm.io/threshold|resize.topolvm.io/threshold>: "20%"
          <http://resize.topolvm.io/increase|resize.topolvm.io/increase>: "25%"
- Ensure Velero’s node-agent (restic) is installed and running. Without it, the annotation has no effect. - You should see a DaemonSet like 
velero-node-agent
(or similar) in the 
velero
namespace and logs without errors. How to verify it’s working 1) Confirm the annotation landed on the StatefulSet pod template and on the pods:
Copy code
kubectl -n prefect get sts -l <http://app.kubernetes.io/name=postgresql|app.kubernetes.io/name=postgresql> -o yaml | yq '.items[0].spec.template.metadata.annotations'
kubectl -n prefect get pod -l <http://app.kubernetes.io/name=postgresql|app.kubernetes.io/name=postgresql> -o yaml | yq '.items[0].metadata.annotations'
2) Confirm the volume name really is `data`:
Copy code
kubectl -n prefect get pod -l <http://app.kubernetes.io/name=postgresql|app.kubernetes.io/name=postgresql> -o jsonpath="{.items[0].spec.volumes[*].name}{'\n'}"
If it’s not 
data
, use whatever name you see in the annotation. 3) Confirm Velero node-agent is present and healthy:
Copy code
kubectl -n velero get ds
kubectl -n velero logs ds/velero-node-agent -c node-agent --tail=200
kubectl -n velero logs deploy/velero --tail=200
4) Run a test backup that includes the 
prefect
namespace and check that the PG pod volume is captured. If you’re using CSI snapshots, restic/node-agent will still honor the annotation for FS backups; otherwise set your Backup spec accordingly (e.g., 
defaultVolumesToFsBackup: true
or 
snapshotVolumes: false
depending on your strategy). Common gotchas - TopoLVM annotations won’t work on 
hcloud-volumes
. Those resize annotations only apply when your storage class is backed by TopoLVM. - If Argo CD applied the change but the pod didn’t roll, tweak the pod template annotation (e.g., add/remove a dummy annotation) to force a rollout, or do a manual restart of the PG StatefulSet. - If using Velero CSI snapshots without node-agent, the 
<http://backup.velero.io/backup-volumes|backup.velero.io/backup-volumes>
annotation won’t be acted on. Install/enable node-agent for file-system backups. References - Prefect Helm charts: https://prefecthq.github.io/prefect-helm - Prefect Server chart (values, dependency on Bitnami PostgreSQL): https://github.com/PrefectHQ/prefect-helm/tree/main/charts/prefect-server - Bitnami PostgreSQL chart docs (values, volume name): https://artifacthub.io/packages/helm/bitnami/postgresql - Velero pod volume backup annotations: https://velero.io/docs/ (see File-system backup / node-agent docs) If you can share: - The rendered StatefulSet for PostgreSQL (
kubectl -n prefect get sts -o yaml
) - The Velero Backup spec and node-agent status/logs …I can pinpoint exactly why the annotation isn’t taking effect in your cluster.
3 Views
Open in Slack
PreviousNext
Powered by