Hey all... I am trying to configure prefect in a k8s cluster. Running into issues though, the docs h...

Adam Brusselback

02/03/2025, 8:05 PM

Hey all... I am trying to configure prefect in a k8s cluster. Running into issues though, the docs here: prefect-server are telling me to put a plain text password in my values.yaml, which is not happening. My postgres cluster already has a secret with the username / password in it, and I was going to hard code the port / host / database in the values. Anyone know the right way to handle this?

Adam Brusselback

02/03/2025, 10:54 PM

The only solution I found so far is to run my helm chart once to create the database cluster / secrets, and then run a helm update so it notices my original secret, and then deploys the following secret:

Copy code

{{- $secret := (lookup "v1" "Secret" .Release.Namespace "goacquire-cluster-prefect-app") }}
{{- if $secret }}
---
apiVersion: v1
kind: Secret
metadata:
  name: prefect-postgres-secret
type: Opaque
data:
  connection-string: {{ printf "<postgresql+asyncpg://%s:%s@goacquire-cluster-prefect-rw:5432/app>" ($secret.data.username | b64dec) ($secret.data.password | b64dec) | b64enc }}
{{- end }}

I really hate this solution... Any other ideas?

Jamie Zieziula

02/03/2025, 11:09 PM

Hi Adam -- the chart supports taking in an existing secret name and loading it to the

prefect-server

Jamie Zieziula

02/03/2025, 11:11 PM

this assumes that the secret is structured as a connection string:

Copy code

<postgresql+asyncpg://username:password@host>:port/db

Jamie Zieziula

02/03/2025, 11:11 PM

does that help?

Mitch Nielsen

02/03/2025, 11:15 PM

I think that section is for basic auth, right? looking lower at https://github.com/PrefectHQ/prefect-helm/blob/main/charts/prefect-server/values.yaml#L500-L504, it looks like there should be

postgresql.auth.existingSecret

- I suspect that's coming from the downstream PostgreSQL chart and we just don't happen to be overriding it from ours

Mitch Nielsen

02/03/2025, 11:17 PM

(source from bitnami's postgresql chart)

Mitch Nielsen

02/03/2025, 11:19 PM

all together should look like this:

Copy code

postgresql:
  auth:
    existingSecret: my-secret
server:
  secret:
    create: false

Adam Brusselback

02/03/2025, 11:19 PM

@Jamie Zieziula That is what I was doing above with that secret... It works, but it requires me running:

Copy code

helm install...
helm update...

before my cluster works

Adam Brusselback

02/03/2025, 11:20 PM

My values.yaml looks like so:

Copy code

prefect-server: 
  secret:
    create: false
    name: prefect-postgres-secret
  postgresql: 
    enabled: false

It does work... I really hate the fact I can't install in one go though.

Jamie Zieziula

02/03/2025, 11:23 PM

What is the error you're seeing from helm?

Jamie Zieziula

02/03/2025, 11:23 PM

And confirming that the secret your referencing contains the full connection string, not just the username/password?

Adam Brusselback

02/03/2025, 11:25 PM

Yeah, my secret contains the full connection string:

Copy code

{{- $secret := (lookup "v1" "Secret" .Release.Namespace "goacquire-cluster-prefect-app") }}
{{- if $secret }}
---
apiVersion: v1
kind: Secret
metadata:
  name: prefect-postgres-secret
type: Opaque
data:
  connection-string: {{ printf "<postgresql+asyncpg://%s:%s@goacquire-cluster-prefect-rw:5432/app>" ($secret.data.username | b64dec) ($secret.data.password | b64dec) | b64enc }}
{{- end }}

Notice the

if

clause there... If I remove it, I get the following error:

Copy code

Error: INSTALLATION FAILED: template: goacquire/templates/prefect-postgres-secret.yaml:12:106: executing "goacquire/templates/prefect-postgres-secret.yaml" at <$secret.data.username>: nil pointer evaluating interface {}.username

Adam Brusselback

02/03/2025, 11:26 PM

because I believe it starts templating before the actual deployment

Adam Brusselback

02/03/2025, 11:27 PM

Sorry, I was working on writing up an issue when you both replied to me: https://github.com/PrefectHQ/prefect-helm/issues/441

Mitch Nielsen

02/03/2025, 11:37 PM

@Adam Brusselback actually if you want to huddle here on Slack, I can catch up on this with you. Otherwise happy to type it out

Mitch Nielsen

02/03/2025, 11:42 PM

• There's no way I know of to control order of operations with Helm. So if you've made Prefect a sub-chart of your own, there's no way to have your chart install, wait for the secret to be created, and then proceed to the Prefect sub-chart templates. • Even if you could control that order, managing secrets in Helm isn't particularly secure anyway. They're not encrypted, only encoded in base64. For production, you're often better off creating a K8s Secret with some other secure mechanism and then only referencing that Secret by name in your helm values. Hope that helps - please let me know if I misunderstood any part(s) of your setup

Adam Brusselback

02/03/2025, 11:50 PM

Yeah, I wasn't wanting to manage secrets with Helm because of the security issues. I have external-secrets setup for some of that already, but I couldn't think of a way to make that work with the Prefect helm chart. Appreciate the info. So I guess that just means my new deploy process needs to be:

Copy code

helm install ...
helm upgrade ...

and I need to live with it.

Adam Brusselback

02/03/2025, 11:51 PM

If you allowed the database config options to be passed in through env vars, easy as cake...but you only allow the full connection string.

Mitch Nielsen

02/03/2025, 11:51 PM

still thinking, but at the moment the only other alternative I'm seeing is to not make prefect a subchart. then you'd

helm install <goacquire

and then

helm install prefect-server

. some tools like

helmfile

make this easier to manage - I wonder if they allow you to specify the order 🤔

Mitch Nielsen

02/03/2025, 11:52 PM

what kind of env var config would you want to pass in? we're always open to contributions/updates

Adam Brusselback

02/03/2025, 11:55 PM

e.g. if you took in

Copy code

prefect-server:
    env:
      - name: POSTGRES_USER
        valueFrom:
          secretKeyRef:
            name: goacquire-cluster-prefect-app
            key: username
      - name: POSTGRES_PASSWORD
        valueFrom:
          secretKeyRef:
            name: goacquire-cluster-prefect-app
            key: password
      - name: POSTGRES_HOST
        value: goacquire-cluster-prefect-rw
      - name: POSTGRES_PORT
        value: 5432
      - name: POSTGRES_DATABASE
        value: app

(or whatever env var names make sense for Prefect). I think that would work at least...if the chart supported it.

Adam Brusselback

02/04/2025, 12:01 AM

Really appreciate the feedback and help by the way.

Mitch Nielsen

02/04/2025, 12:04 AM

I think we could add support for

extraEnv

and allow you to provide a list of map entries where you can specify all that. if you're interested in opening a PR for that we'd be happy to review it, otherwise we can try to get to that soon edit: actually, this would require an application change, too, to accept individual env vars and parse together the connection string itself rather than the existing all-in-one

PREFECT_API_DATABASE_CONNECTION_URL

. I can look into that more on the app side and see if that's supported, and if not we can propose it

Mitch Nielsen

02/04/2025, 12:04 AM

sure thing, thanks for reaching out about it

Mitch Nielsen

02/04/2025, 12:06 AM

(edited my message above btw)

Adam Brusselback

02/04/2025, 12:08 AM

Got it. That makes sense. I have my workaround for now, so i'll go with that until I see something change in the release notes 😉 Thanks again!

Mitch Nielsen

02/04/2025, 12:51 AM

you're welcome!

22 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.