https://prefect.io logo
#prefect-server
Title
# prefect-server
s

Sam Werbalowsky

09/15/2021, 9:57 PM
I am trying to run a simple flow to validate that I have my gitlab storage configured correctly. I’ve deployed Prefect Server using the helm chart on Kubernetes, with the agent. I am getting an error that the secret is not found…where/how should I be setting the secret? I had tried exec-ing into the agent and setting it via export but that doesn’t seem to work. That wouldn’t solve the issue of pulling secrets somewhere automatically in any case. I don’t want to set the secret in the flow, since it would be sensitive information. How can I set them via helm or a different way?
g

Gaylord Cherencey

09/15/2021, 10:17 PM
Hey @Sam Werbalowsky personally I did it this way by running the agent with env variable
prefect agent kubernetes install -t $1 --env PREFECT__CONTEXT__SECRETS__GITHUB=$2 --rbac | kubectl apply -f -
then on the Github Storage you need to pass the secret environment name
Copy code
flow.storage = GitHub(
    repo="org/repo",
    path="flows/my_flow.py",
    access_token_secret="GITHUB"
)
s

Sam Werbalowsky

09/15/2021, 10:18 PM
Interesting, ok...so you just run that from the local command line or have it as part of CI?
k

Kevin Kho

09/15/2021, 10:20 PM
I think Gaylord is right here. It would be upon agent spin up.
s

Sam Werbalowsky

09/15/2021, 10:24 PM
Cool, thanks...Will try tomorrow. So tldr add secrets to agent on spin up.
Can confirm this works. It would be nice if there were a way in the helm chart to set secrets, since they have to be set on the agent…I believe our alternative is probably to modify the helm chart or the template for agent somehow.
k

Kevin Kho

09/17/2021, 4:48 PM
If you do what Gaylord did, it will add the env variables for you in the agent job template right? You are right you could do that manually too if that’s what you are suggesting by editing the job template
s

Sam Werbalowsky

09/17/2021, 4:56 PM
yes, correct - it feels like Secrets are not the easiest to work with with server, that’s all.
i.e. using kubeseal or kubernetes secrets would be a nice feature
k

Kevin Kho

09/17/2021, 5:03 PM
There is a community contributed KubernetesSecret task that might help you?
🙏 1
👀 1
c

Chris L.

09/18/2021, 1:29 PM
Hey @Sam Werbalowsky I’m a bit late into the thread but just wanted to share a point about K8 secrets and Prefect. So, instead of using “—env” flags with prefect agent install, I tried to pass K8 secrets to directly pass secrets into my flows as environment variables. I ended up pulling the Prefect server helm chart and adding my secrets as “secret” env vars (with valueFrom and secretKeyRef) under the “agent” container in this file https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml The pros of having the whole chart in your project is the ability to version control the secrets in your flow via this file and any k8 secrets templates you add to the Helm chart. I think it’s worth moving to this setup instead of writing glue to get Prefect secrets and contexts to work well with your K8 secrets (if K8’s secrets management is what you want at the end of the day). EDIT: The suggestion above does not work. User-defined env vars in Prefect Agent's
deployments.yml
do NOT pass into the K8
job
(which deploys and runs the flow). What should be modified is the
prefect/agent/kubernetes/job_template.yml
file (coupled with
KubernetesRun
). I believe it should be possible to add env vars that to refer K8 secrets (as suggested above) to
job_template.yml
. You then COPY this job_template.yml into `KubernetesAgent`'s image. Finally, you either set
job_template_path
in `KubernetesRun`or modify line 44 in
deployment.yml
from
prefect agent kubernetes start
to
prefect agent kubernetes start --job-template path/to/my_template.yaml
I found that the inline comments for
prefect/agent/kubernetes/agent.py
explain how
KubernetesAgent
sets env vars for the k8 job which runs the flow:
Copy code
# Populate environment variables from the following sources,
        # with precedence:
        # - Values required for flow execution, hardcoded below
        # - Values set on the KubernetesRun object
        # - Values set using the `--env` CLI flag on the agent
        # - Values in the job template
References: https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template Sorry for the trouble! I'll try out the recipe above on Sunday and report what I find.
s

Sam Werbalowsky

09/18/2021, 2:05 PM
Awesome, thank you!