https://prefect.io logo
Title
j

Jakub Vedral

01/29/2021, 8:20 AM
Hi there, is there a way how to add Secrets to already started prefect server (prefect core)? or to check somehow that the secrets are present there?
a

Amanda Wee

01/29/2021, 8:40 AM
In my case, I needed to set the
SLACK_WEBHOOK_URL
secret, and did so by setting a
PREFECT__CONTEXT__SECRETS__SLACK_WEBHOOK_URL
environment variable on the Docker container (to be used in ECS, but also for local testing) running the prefect local agent, and it worked. I was also able to check the secret like this:
def slack_notifications_enabled() -> bool:
    """Return True if Slack notifications are enabled."""
    slack_webhook_url = Secret("SLACK_WEBHOOK_URL")
    return slack_webhook_url.exists()
and use the above code to decide whether or not to set a state handler on the flow (so I can run the flow locally without sending a Slack notification). The prefect server setup was not provided with the secret.
j

Jakub Vedral

01/29/2021, 8:46 AM
Thanks for a quick reply. Did you set the ENV variable using config.toml or docker-compose file? I dont know where to find docker-compose file or to tell prefect server to modify it with my secrets. I wonder how you managed to set it before the start of prefect server containers... 🙂
a

Amanda Wee

01/29/2021, 8:59 AM
Neither. It was set on the Docker container itself, but in my case I'm doing both flow registration and running the local agent from the same shell script that the Docker container runs. So my guess is that you would set that env var in the context of the flow registration script if you're going to access the secret outside of a flow, and in the context of the
prefect agent local start
if you are going to access it within the flow. I think using the config file might work too, but I have not tried it. I did not set it before the start of the prefect server, but rather after they were started.
j

Jakub Vedral

01/29/2021, 9:36 AM
Ok, i have to try that --env option with the agent. I still have a feeling that there should be some easier way to set them once and for good (besides the cloud plan) 🙂
still wonder what is the preferred way to add new secrets to already running prefect core server ... But given that I have only one agent, it would be probably best to set the ENV vars on it to define those secrets.
j

Jeremiah

01/29/2021, 2:06 PM
Hi @Jakub Vedral - when using Prefect Server, all secrets are managed locally by the flow, so there’s no need to provide them to the server itself (via docker-compose or otherwise)
If you are using Prefect Cloud, there’s an API for setting secrets on the server so that they can be sent to flows automatically. Here are docs explaining the Cloud API and also how to set them locally for Server: https://docs.prefect.io/orchestration/concepts/secrets.html
j

Jakub Vedral

01/29/2021, 3:14 PM
Hi, "all secrets are managed locally by the flow" - does it mean that i have to define them only in the flow during registration by method
client.set_secret(name="my secret", value=42)
? I guess GraphQL option is only for the Cloud version aswell. I was wondering if setting them as ENV variables on agent (using the --env flag) is the way to keep them persistent and share them among several flows.
j

Jeremiah

01/29/2021, 3:17 PM
client.set_secret
is calling Prefect Cloud’s secret manager API, so it won’t work with Prefect Server. However, setting env vars is a good way to configure and provide information to any flow run launched by that agent
🙌 1
j

Jakub Vedral

01/30/2021, 8:48 AM
Thank you for help. ENV variables on agent machine are probably the best bet.