Hey everyone, I’ve got some

secrets

that I need to use in most of my tasks in a flow. I hate the visual of an upstream dependency to the same secret task from almost all my tasks. Is there a way for custom secrets to behave like, say

AWS_CREDENTIALS

, as per: https://docs.prefect.io/core/concepts/secrets.html#default-secrets Or a prefect backend compatible way of having my secrets reside in

prefect.context

, or a similar context like construct?

I’m new to prefect, so unsure if this is the way to go, but I just use:

postgres_secret = prefect.context.secrets.get('POSTGRES')

For which I provide the env as

export PREFECT__CONTEXT__SECRETS__POSTGRES='{
    "database": "",
    "username": "",
    "host": "",
    "port": 5432,
    "password": ""
}'

Thank you, but I want to avoid setting my secrets in environment variables. This would require me to store my secrets inside my docker storage as environment variables. I want to access secrets only on runtime.

Alright, I’m unsure, but I dont think it matters how you retrieve the secret.

from prefect.client.secrets import Secret
s = Secret("POSTGRES")
print(s.get())

That also works for me, so prefect does store it as a secret, regardless of provided name. Or maybe I’m just missing what you want to achive exactly (again prefect is new to me 😅)

It’s not that I really care how I retrieve it, its more about where it is stored. Think about when you want to deploy your flow somehow, so that it is ready to run. If I use envvars for prefect secrets, they are stored inside the docker image. Anyone with access to the docker registry will be able to pull the image and run

docker inspect

to find out any secrets. I actually want my secrets to be read at flow run time, or just before flow run time. I definitely do not want my secrets in the docker image, for the aforementioned case. What I have in mind is storing my secrets in AWS Secrets Manager. Production machines on AWS will have permission to retrieve secrets from the secret manager. However, people with access to the docker image still can’t retrieve secrets, since they don’t have access to the secret manager.

Ah I see, I haven’t got around deploying Prefect personally, so maybe thats the reason for our miscomms. Personally, I’ll probably deploy Prefect in Kubernetes and then supply the secrets as environment variables, hence my reasoning. I hope someone else can help you then 🙂

Thanks 🙂 talking it through gave me some hacky ideas, I’ll see if those work. I should also learn k8, its due time 😅

Hi @emre - are you using Prefect Cloud? If so, you can store secrets in Vault and access them directly from your tasks (which would avoid the dependency ugliness you mentioned earlier). Otherwise I don't think there's a straightforward way of storing secrets without a persistence layer.

ah nice, then this is a matter of updating the prefect ui 😅

Sorry @Julian I'm not sure I follow, is there something missing from the UI?

ups sorry, wrong thread, this was related to https://prefect-community.slack.com/archives/CL09KU1K7/p1604566708069300

@nicholas We are currently on prefect core / server sadly. Thanks for the update!

Hi @emre i had same idea. I found that you can pass secret to your context during start of your agent - https://github.com/PrefectHQ/prefect/blob/0e9fa02c7acac586610c9894bec82e90fdf57c8b/docs/orchestration/recipes/third_party_auth.md#passing-secrets-from-an-agent. Or you can probably do something like this:

with Flow("secret") as f:
   prefect.context.secrets["MY_KEY"] = func_to_get_secret_from_aws()

I am going to use Hashicorp Vault for storing secrets. So I am probably going to store auth key to Vault in Agent. And then in every Flow I will ask for the secrets with some custom function and store result to context

Thanks @Radek Tomsej Setting the context secrets within python has been working great for our core only runs. So great that I feel a little sad it isn’t available for

server

runs. Passing secrets to agents would work well, I will keep it in mind. Although I would feel better if the secrets were coupled with the flow that needs them, rather than the agent. I will first try to retrieve and set secrets with a docker entrypoint, if that doesn’t work, passing secrets from an agent will definitely do well.