<@ULVA73B9P> can I specify an initcontainer in my ...
# ask-marvin
s
@Marvin can I specify an initcontainer in my prefect server deployment (via kubernetes helm chart)?
m
thought for 259.3 seconds
Short answer: yes — you can add initContainers to the Prefect Server pods via the Helm chart. Where to put it - The chart exposes initContainer hooks for both the API and UI pods. Depending on your exact chart version, the keys will be one of these common shapes: - serverApi.initContainers / ui.initContainers - or nested under serverApi.pod.initContainers / ui.pod.initContainers Quick way to confirm the exact keys for your version (2025.6.4170433):
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update
helm show values prefect/prefect-server --version 2025.6.4170433 | less
Example values snippets Option A (flat under serverApi):
Copy code
serverApi:
  # Add an initContainer (e.g., wait for a secret or DB to be ready)
  initContainers:
    - name: wait-for-db
      image: postgres:16
      command: ["sh","-c"]
      args:
        - |
          echo "Waiting for Postgres...";
          until pg_isready -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER"; do
            sleep 3
          done
      env:
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: my-db-secret
              key: host
        - name: DB_PORT
          value: "5432"
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: my-db-secret
              key: username

  # (Optional) extra volumes if you need to mount secrets/files
  extraVolumes:
    - name: db-secret
      secret:
        secretName: my-db-secret
  extraVolumeMounts:
    - name: db-secret
      mountPath: /var/run/db

  # Pick up DB URL from the ExternalSecrets-managed Secret
  env:
    - name: PREFECT_API_DATABASE_CONNECTION_URL
      valueFrom:
        secretKeyRef:
          name: my-db-secret
          key: url
Option B (nested under pod):
Copy code
serverApi:
  pod:
    initContainers:
      - name: wait-for-db
        image: postgres:16
        command: ["sh","-c"]
        args:
          - |
            echo "Waiting for Postgres...";
            until pg_isready -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER"; do
              sleep 3
            done
        env:
          - name: DB_HOST
            valueFrom:
              secretKeyRef:
                name: my-db-secret
                key: host
          - name: DB_PORT
            value: "5432"
          - name: DB_USER
            valueFrom:
              secretKeyRef:
                name: my-db-secret
                key: username

    annotations:
      <http://reloader.stakater.com/auto|reloader.stakater.com/auto>: "true"  # see rotation note below

  env:
    - name: PREFECT_API_DATABASE_CONNECTION_URL
      valueFrom:
        secretKeyRef:
          name: my-db-secret
          key: url
Keeping Prefect running when your Postgres password rotates - Kubernetes does not update environment variables in a running pod when a Secret changes. Since Prefect Server reads the DB URL at startup, you’ll need a restart to pick up the new password. - Recommended pattern: add an automatic restart mechanism that watches your Secret: - Stakater Reloader operator: annotate the Deployment/pod template so pods restart when the referenced Secret changes. - Add:
<http://reloader.stakater.com/auto|reloader.stakater.com/auto>: "true"
- Reloader: https://github.com/stakater/Reloader - With External Secrets Operator (ESO), your AWS Secret updates -> ESO refreshes the Kubernetes Secret -> Reloader restarts the Prefect API pods -> Prefect picks up the new password on startup. Useful links - Prefect Helm repo: github.com/PrefectHQ/prefect-helm - Helm repo index: prefecthq.github.io/prefect-helm - Prefect 3.x Server on Helm docs: docs-3.prefect.io/v3/advanced/server-helm A few questions to tailor this precisely:
- Are you aiming for the initContainer to just wait for the ESO-managed Secret to exist, or to actively verify DB connectivity before the API starts? - What’s the name/keys of the K8s Secret ESO writes for your DB URL? I can adapt the snippet exactly to your setup once you share that.
s
@Marvin I don't see initcontainers but I see extraContainers
m
thought for 196.8 seconds
You’re right — in the current Prefect Server Helm chart, you’ll see
extraContainers
but not an
initContainers
field. That means the chart doesn’t natively expose initContainers today. Extra containers run alongside the main container, so they won’t block startup like a true initContainer. Good alternatives depending on your goal: - If your goal is secret/password rotation (with External Secrets Operator): - You don’t need an initContainer. Instead, ensure the API pods restart when the K8s Secret changes so they pick up the new DB URL/password at startup. - Common approach: Stakater Reloader. Annotate the API deployment via pod annotations so it restarts on secret updates:
Copy code
serverApi:
      pod:
        annotations:
          <http://reloader.stakater.com/auto|reloader.stakater.com/auto>: "true"
Reloader: https://github.com/stakater/Reloader - If your goal is to block API startup until Postgres is reachable: - If your chart version exposes command/args overrides for the API container (sometimes
serverApi.command
/
serverApi.args
), you can wrap the entrypoint to wait on
pg_isready
before starting:
Copy code
serverApi:
      command: ["sh","-c"]
      args:
        - |
          until pg_isready -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER"; do
            echo "Waiting for Postgres..."; sleep 3;
          done
          exec prefect server start --host 0.0.0.0 --port 4200
      env:
        - name: DB_HOST
          valueFrom:
            secretKeyRef: { name: my-db-secret, key: host }
        - name: DB_PORT
          value: "5432"
        - name: DB_USER
          valueFrom:
            secretKeyRef: { name: my-db-secret, key: username }
        - name: PREFECT_API_DATABASE_CONNECTION_URL
          valueFrom:
            secretKeyRef: { name: my-db-secret, key: url }
- If command/args aren’t exposed, you can still get a true initContainer via a Helm post-renderer (e.g., kustomize/yq) to inject
spec.template.spec.initContainers
into the rendered Deployment: - Render with a post-renderer script that adds:
Copy code
spec:
        template:
          spec:
            initContainers:
              - name: wait-for-db
                image: postgres:16
                command: ["sh","-c"]
                args:
                  - |
                    until pg_isready -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER"; do
                      echo "Waiting for Postgres..."; sleep 3;
                    done
                env:
                  - name: DB_HOST
                    valueFrom:
                      secretKeyRef: { name: my-db-secret, key: host }
                  - name: DB_PORT
                    value: "5432"
                  - name: DB_USER
                    valueFrom:
                      secretKeyRef: { name: my-db-secret, key: username }
- Or use a policy engine (Kyverno/OPA) to mutate the pod and inject an initContainer. To double-check what your chart version supports, you can inspect the values for your exact version (you mentioned 2025.6.4170433):
Copy code
helm repo add prefect <https://prefecthq.github.io/prefect-helm>
helm repo update
helm show values prefect/prefect-server --version 2025.6.4170433 | less
Links - Prefect Helm charts repo: github.com/PrefectHQ/prefect-helm - Helm repo index: prefecthq.github.io/prefect-helm - Prefect 3.x Server on Helm docs: docs-3.prefect.io/v3/advanced/server-helm If you share the output snippet around
serverApi
from your
helm show values
, I can tailor the exact values block (annotations vs command/args) for your chart. What’s the specific behavior you want the “init” step to handle?
s
@Marvin I want to mount a .pgpass file from a kubernetes secret but I need to change the permissions on the file to 0600
m
thought for 63.1 seconds
Great news: you don’t need an initContainer just to set 0600 on a .pgpass file. You can set file permissions directly on a Secret volume with defaultMode. Simplest approach (works if the Prefect API container can read files owned by root, e.g., running as root): - Mount the Secret as a volume and set defaultMode to 0600 - Mount it at the desired path (e.g., /home/prefect/.pgpass) - Optionally set PGPASSFILE so libpq knows where to look Example values.yaml snippet (paths may vary by chart version; many expose extraVolumes/extraVolumeMounts under serverApi):
Copy code
serverApi:
  extraVolumes:
    - name: pgpass
      secret:
        secretName: my-pgpass-secret
        # 0600 in decimal is 384; YAML 1.2 may not accept 0600 octal literal
        defaultMode: 384
        items:
          - key: pgpass
            path: .pgpass
  extraVolumeMounts:
    - name: pgpass
      mountPath: /home/prefect
      readOnly: true
  env:
    - name: PGPASSFILE
      value: /home/prefect/.pgpass
Notes: - defaultMode: 384 is 0600. If your tooling supports it, 0o600 can also be used: - defaultMode: 384 (recommended) - or defaultMode: 0o600 (if your YAML parser supports it) - If your container runs as non-root: - A 0600 file owned by root will not be readable by a non-root user. In that case, you’ll need to write the file with the same UID/GID as the main container. - Since the chart doesn’t expose initContainers, you can: - Use an emptyDir + extraContainers sidecar that writes the file with umask 177 (0600) and matching runAsUser/runAsGroup, then mount that emptyDir into the main container. - Or inject a true initContainer with a Helm post-renderer (kustomize/kyverno). Example using a sidecar to write the file (for non-root main container):
Copy code
serverApi:
  pod:
    securityContext:
      runAsUser: 1000
      runAsGroup: 1000
      fsGroup: 1000

  # Source: Secret -> /src/pgpass
  extraVolumes:
    - name: pgpass-src
      secret:
        secretName: my-pgpass-secret
        items:
          - key: pgpass
            path: pgpass
    - name: pgpass
      emptyDir: {}

  extraContainers:
    - name: write-pgpass
      image: busybox:1.36
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
      command:
        - sh
        - -c
        - |
          umask 177
          cp /src/pgpass /dst/.pgpass
          # keep running so the file remains available
          tail -f /dev/null
      volumeMounts:
        - name: pgpass-src
          mountPath: /src
        - name: pgpass
          mountPath: /dst

  extraVolumeMounts:
    - name: pgpass
      mountPath: /home/prefect

  env:
    - name: PGPASSFILE
      value: /home/prefect/.pgpass
If this .pgpass is specifically for Prefect Server’s DB connection, you can also avoid .pgpass entirely by setting the database URL via Secret-backed env var:
Copy code
serverApi:
  env:
    - name: PREFECT_API_DATABASE_CONNECTION_URL
      valueFrom:
        secretKeyRef:
          name: my-db-secret
          key: url
This is usually simpler and avoids file permission concerns. With External Secrets + Stakater Reloader, the pods will restart on secret changes and pick up new credentials. Links - Prefect Helm charts: github.com/PrefectHQ/prefect-helm - Helm repo index: prefecthq.github.io/prefect-helm - Reloader (for secret-driven restarts): https://github.com/stakater/Reloader Do you know if your Prefect API container runs as root or non-root? I can tailor the snippet accordingly.