I am trying to run a simple flow to validate that I have my gitlab storage configured correctly. I’ve deployed Prefect Server using the helm chart on Kubernetes, with the agent. I am getting an error that the secret is not found…where/how should I be setting the secret? I had tried exec-ing into the agent and setting it via export but that doesn’t seem to work. That wouldn’t solve the issue of pulling secrets somewhere automatically in any case. I don’t want to set the secret in the flow, since it would be sensitive information. How can I set them via helm or a different way?

Hey @Sam Werbalowsky personally I did it this way by running the agent with env variable

prefect agent kubernetes install -t $1 --env PREFECT__CONTEXT__SECRETS__GITHUB=$2 --rbac | kubectl apply -f -

then on the Github Storage you need to pass the secret environment name

flow.storage = GitHub(
    repo="org/repo",
    path="flows/my_flow.py",
    access_token_secret="GITHUB"
)

Interesting, ok...so you just run that from the local command line or have it as part of CI?

I think Gaylord is right here. It would be upon agent spin up.

Cool, thanks...Will try tomorrow. So tldr add secrets to agent on spin up.

If you do what Gaylord did, it will add the env variables for you in the agent job template right? You are right you could do that manually too if that’s what you are suggesting by editing the job template

yes, correct - it feels like Secrets are not the easiest to work with with server, that’s all.

There is a community contributed KubernetesSecret task that might help you?

Hey @Sam Werbalowsky I’m a bit late into the thread but just wanted to share a point about K8 secrets and Prefect. So, instead of using “—env” flags with prefect agent install, I tried to pass K8 secrets to directly pass secrets into my flows as environment variables. I ended up pulling the Prefect server helm chart and adding my secrets as “secret” env vars (with valueFrom and secretKeyRef) under the “agent” container in this file https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml The pros of having the whole chart in your project is the ability to version control the secrets in your flow via this file and any k8 secrets templates you add to the Helm chart. I think it’s worth moving to this setup instead of writing glue to get Prefect secrets and contexts to work well with your K8 secrets (if K8’s secrets management is what you want at the end of the day). EDIT: The suggestion above does not work. User-defined env vars in Prefect Agent's

deployments.yml

do NOT pass into the K8

job

(which deploys and runs the flow). What should be modified is the

prefect/agent/kubernetes/job_template.yml

file (coupled with

KubernetesRun

). I believe it should be possible to add env vars that to refer K8 secrets (as suggested above) to

job_template.yml

. You then COPY this job_template.yml into `KubernetesAgent`'s image. Finally, you either set

job_template_path

in `KubernetesRun`or modify line 44 in

deployment.yml

from

prefect agent kubernetes start

to

prefect agent kubernetes start --job-template path/to/my_template.yaml

I found that the inline comments for

prefect/agent/kubernetes/agent.py

explain how

KubernetesAgent

sets env vars for the k8 job which runs the flow:

# Populate environment variables from the following sources,
        # with precedence:
        # - Values required for flow execution, hardcoded below
        # - Values set on the KubernetesRun object
        # - Values set using the `--env` CLI flag on the agent
        # - Values in the job template

References: https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template Sorry for the trouble! I'll try out the recipe above on Sunday and report what I find.

Awesome, thank you!