emre

    emre

    1 year ago
    Hey everyone, I’ve got some
    secrets
    that I need to use in most of my tasks in a flow. I hate the visual of an upstream dependency to the same secret task from almost all my tasks. Is there a way for custom secrets to behave like, say
    AWS_CREDENTIALS
    , as per: https://docs.prefect.io/core/concepts/secrets.html#default-secrets Or a prefect backend compatible way of having my secrets reside in
    prefect.context
    , or a similar context like construct?
    Joël Luijmes

    Joël Luijmes

    1 year ago
    I’m new to prefect, so unsure if this is the way to go, but I just use:
    postgres_secret = prefect.context.secrets.get('POSTGRES')
    For which I provide the env as
    export PREFECT__CONTEXT__SECRETS__POSTGRES='{
        "database": "",
        "username": "",
        "host": "",
        "port": 5432,
        "password": ""
    }'
    emre

    emre

    1 year ago
    Thank you, but I want to avoid setting my secrets in environment variables. This would require me to store my secrets inside my docker storage as environment variables. I want to access secrets only on runtime.
    Joël Luijmes

    Joël Luijmes

    1 year ago
    Alright, I’m unsure, but I dont think it matters how you retrieve the secret.
    from prefect.client.secrets import Secret
    s = Secret("POSTGRES")
    print(s.get())
    That also works for me, so prefect does store it as a secret, regardless of provided name. Or maybe I’m just missing what you want to achive exactly (again prefect is new to me 😅)
    emre

    emre

    1 year ago
    It’s not that I really care how I retrieve it, its more about where it is stored. Think about when you want to deploy your flow somehow, so that it is ready to run. If I use envvars for prefect secrets, they are stored inside the docker image. Anyone with access to the docker registry will be able to pull the image and run
    docker inspect
    to find out any secrets. I actually want my secrets to be read at flow run time, or just before flow run time. I definitely do not want my secrets in the docker image, for the aforementioned case. What I have in mind is storing my secrets in AWS Secrets Manager. Production machines on AWS will have permission to retrieve secrets from the secret manager. However, people with access to the docker image still can’t retrieve secrets, since they don’t have access to the secret manager.
    Joël Luijmes

    Joël Luijmes

    1 year ago
    Ah I see, I haven’t got around deploying Prefect personally, so maybe thats the reason for our miscomms. Personally, I’ll probably deploy Prefect in Kubernetes and then supply the secrets as environment variables, hence my reasoning. I hope someone else can help you then 🙂
    emre

    emre

    1 year ago
    Thanks 🙂 talking it through gave me some hacky ideas, I’ll see if those work. I should also learn k8, its due time 😅
    nicholas

    nicholas

    1 year ago
    Hi @emre - are you using Prefect Cloud? If so, you can store secrets in Vault and access them directly from your tasks (which would avoid the dependency ugliness you mentioned earlier). Otherwise I don't think there's a straightforward way of storing secrets without a persistence layer.
    j

    Julian

    1 year ago
    ah nice, then this is a matter of updating the prefect ui 😅
    nicholas

    nicholas

    1 year ago
    Sorry @Julian I'm not sure I follow, is there something missing from the UI?
    j

    Julian

    1 year ago
    emre

    emre

    1 year ago
    @nicholas We are currently on prefect core / server sadly. Thanks for the update!
    Radek Tomsej

    Radek Tomsej

    1 year ago
    Hi @emre i had same idea. I found that you can pass secret to your context during start of your agent - https://github.com/PrefectHQ/prefect/blob/0e9fa02c7acac586610c9894bec82e90fdf57c8b/docs/orchestration/recipes/third_party_auth.md#passing-secrets-from-an-agent. Or you can probably do something like this:
    with Flow("secret") as f:
       prefect.context.secrets["MY_KEY"] = func_to_get_secret_from_aws()
    I am going to use Hashicorp Vault for storing secrets. So I am probably going to store auth key to Vault in Agent. And then in every Flow I will ask for the secrets with some custom function and store result to context
    emre

    emre

    1 year ago
    Thanks @Radek Tomsej Setting the context secrets within python has been working great for our core only runs. So great that I feel a little sad it isn’t available for
    server
    runs. Passing secrets to agents would work well, I will keep it in mind. Although I would feel better if the secrets were coupled with the flow that needs them, rather than the agent. I will first try to retrieve and set secrets with a docker entrypoint, if that doesn’t work, passing secrets from an agent will definitely do well.