https://prefect.io logo
Powered by
Title
u

2j

10/06/2021, 2:27 AM
Hi all, is there a preferred way to pass along secrets / env vars to the helm deployment of prefect-server during a CI/CD pipeline? For more context, I'm trying to give the deployment a 
AZURE_STORAGE_CONNECTION_STRING
for access to blob storage. I'm not seeing anything baked into the helm values.yaml (though maybe something in 
prefectConfig
could work?). I was thinking of attaching a k8s secret after the helm deployment? But I think it needs to be set on the job that the agent kicks off. And I should be able to set the env var as-is, or in secret form like 
PREFECT__CONTEXT__SECRETS__AZURE_STORAGE_CONNECTION_STRING
?
More info: setting the secret in the agent doesn't seem to work b/c it doesn't pass it on to the job that actually runs the flow. I think I might need to modify the helm chart, or somehow pass it with 
-e
on https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml#L50
t

Thomas Fredriksen

10/06/2021, 5:57 AM
Not what best practice is, but I can share what we did to solve this for our Prefect Server deployment
We have a factory-implementation of a secret-backend which will read secrets based on what environment the task is running in. When developing and testing the flow locally on the develop machine, the tasks will be handed a client which reads from the environment variables. While running in the Prefect Server, the backend will read kubernetes-secrets
We use this implementation to load the credentials to our Azure Vault, and have implemented a task which will read and parse secrets from there.
Most secrets will be passed as task-arguments, so using the Vault-Implementation for this works really well. However, the vault-credentials need to be kept somewhere too, that's where the secret-backend is required in order to hard-code these credentials or provide them when constructing the flow (inside 
with Flow(...) as flow
for example).
Another way of achieving this is by setting the secret as an environment variable in the 
job_config.yaml
that is with the agent. The Helm-chart does not have this built in, so you would need to disable the agent in the helm-chart and create your own kubernetes deployment with a config-map holding the 
job_config.yaml
.
The deployment is available here: https://github.com/PrefectHQ/prefect/tree/master/src/prefect/agent/kubernetes
note that this also creates a rbac role and role binding for the agent as well
The 
job_template.yaml
here is quite sparse, but the default template in the Helm-chart is nice, however the empty volume mounts caused an issue for us. Instead, here is a template that will work: 
apiVersion: batch/v1
kind: Job
metadata:
  name: prefect-job
  labels: {}
spec:
  template:
    metadata:
      labels: {}
    spec:
      containers:
        - name: flow
          image: prefecthq/prefect:latest
          imagePullPolicy: IfNotPresent
          args:
            - prefect
            - execute
            - flow-run
          env:
            - name: PREFECT__CLOUD__API
              value: PREFECT__CLOUD__API
            - name: PREFECT__CLOUD__AUTH_TOKEN
              value: PREFECT__CLOUD__AUTH_TOKEN
            - name: PREFECT__CONTEXT__FLOW_RUN_ID
              value: PREFECT__CONTEXT__FLOW_RUN_ID
            - name: PREFECT__CONTEXT__FLOW_ID
              value: PREFECT__CONTEXT__FLOW_ID
            - name: PREFECT__CONTEXT__NAMESPACE
              value: PREFECT__CONTEXT__NAMESPACE
            - name: PREFECT__CLOUD__AGENT__LABELS
              value: PREFECT__CLOUD__AGENT__LABELS
            - name: PREFECT__CLOUD__SEND_FLOW_RUN_LOGS
              value: PREFECT__CLOUD__SEND_FLOW_RUN_LOGS
            # Backwards compatibility variable for `SEND_FLOW_RUN_LOGS`
            - name: PREFECT__LOGGING__LOG_TO_CLOUD
              value: PREFECT__LOGGING__LOG_TO_CLOUD
            - name: PREFECT__LOGGING__LEVEL
              value: "INFO"
            - name: PREFECT__CLOUD__USE_LOCAL_SECRETS
              value: "false"
            - name: PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS
              value: "prefect.engine.cloud.CloudFlowRunner"
            - name: PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS
              value: "prefect.engine.cloud.CloudTaskRunner"
            # Custom stuff
            - name: CUSTOM__AZ_STORAGE_CONNECTION_STRING
              value: "MY CONNECTION STRING"
          resources:
            requests:
              cpu: "100m"
            limits:
              cpu: "100m"
      restartPolicy: Never
u

2j

10/06/2021, 1:33 PM
@Thomas Fredriksen thank you so much for sharing your strategy! Now I have some more playing around to do
#prefect-serverJoin Slack
Powered by