s

    Sam Werbalowsky

    1 year ago
    I am trying to run a simple flow to validate that I have my gitlab storage configured correctly. I’ve deployed Prefect Server using the helm chart on Kubernetes, with the agent. I am getting an error that the secret is not found…where/how should I be setting the secret? I had tried exec-ing into the agent and setting it via export but that doesn’t seem to work. That wouldn’t solve the issue of pulling secrets somewhere automatically in any case. I don’t want to set the secret in the flow, since it would be sensitive information. How can I set them via helm or a different way?
    Gaylord Cherencey

    Gaylord Cherencey

    1 year ago
    Hey @Sam Werbalowsky personally I did it this way by running the agent with env variable
    prefect agent kubernetes install -t $1 --env PREFECT__CONTEXT__SECRETS__GITHUB=$2 --rbac | kubectl apply -f -
    then on the Github Storage you need to pass the secret environment name
    flow.storage = GitHub(
        repo="org/repo",
        path="flows/my_flow.py",
        access_token_secret="GITHUB"
    )
    s

    Sam Werbalowsky

    1 year ago
    Interesting, ok...so you just run that from the local command line or have it as part of CI?
    Kevin Kho

    Kevin Kho

    1 year ago
    I think Gaylord is right here. It would be upon agent spin up.
    s

    Sam Werbalowsky

    1 year ago
    Cool, thanks...Will try tomorrow. So tldr add secrets to agent on spin up.
    Can confirm this works. It would be nice if there were a way in the helm chart to set secrets, since they have to be set on the agent…I believe our alternative is probably to modify the helm chart or the template for agent somehow.
    Kevin Kho

    Kevin Kho

    1 year ago
    If you do what Gaylord did, it will add the env variables for you in the agent job template right? You are right you could do that manually too if that’s what you are suggesting by editing the job template
    s

    Sam Werbalowsky

    1 year ago
    yes, correct - it feels like Secrets are not the easiest to work with with server, that’s all.
    i.e. using kubeseal or kubernetes secrets would be a nice feature
    Kevin Kho

    Kevin Kho

    1 year ago
    There is a community contributed KubernetesSecret task that might help you?
    Chris L.

    Chris L.

    1 year ago
    Hey @Sam Werbalowsky I’m a bit late into the thread but just wanted to share a point about K8 secrets and Prefect. So, instead of using “—env” flags with prefect agent install, I tried to pass K8 secrets to directly pass secrets into my flows as environment variables. I ended up pulling the Prefect server helm chart and adding my secrets as “secret” env vars (with valueFrom and secretKeyRef) under the “agent” container in this file https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml The pros of having the whole chart in your project is the ability to version control the secrets in your flow via this file and any k8 secrets templates you add to the Helm chart. I think it’s worth moving to this setup instead of writing glue to get Prefect secrets and contexts to work well with your K8 secrets (if K8’s secrets management is what you want at the end of the day). EDIT: The suggestion above does not work. User-defined env vars in Prefect Agent's
    deployments.yml
    do NOT pass into the K8
    job
    (which deploys and runs the flow). What should be modified is the
    prefect/agent/kubernetes/job_template.yml
    file (coupled with
    KubernetesRun
    ). I believe it should be possible to add env vars that to refer K8 secrets (as suggested above) to
    job_template.yml
    . You then COPY this job_template.yml into KubernetesAgent's image. Finally, you either set
    job_template_path
    in KubernetesRunor modify line 44 in
    deployment.yml
    from
    prefect agent kubernetes start
    to
    prefect agent kubernetes start --job-template path/to/my_template.yaml
    I found that the inline comments for
    prefect/agent/kubernetes/agent.py
    explain how
    KubernetesAgent
    sets env vars for the k8 job which runs the flow:
    # Populate environment variables from the following sources,
            # with precedence:
            # - Values required for flow execution, hardcoded below
            # - Values set on the KubernetesRun object
            # - Values set using the `--env` CLI flag on the agent
            # - Values in the job template
    References: https://docs.prefect.io/orchestration/agents/kubernetes.html#custom-job-template Sorry for the trouble! I'll try out the recipe above on Sunday and report what I find.
    s

    Sam Werbalowsky

    1 year ago
    Awesome, thank you!