j

    Jason Nochlin

    2 years ago
    Quick question on best practices: I'm trying to use "Secrets" with a ShellTask and see two possibilities: 1. Get the Secret when I register the flow, eg:
    from prefect import Flow
    from prefect.tasks.shell import ShellTask
    from prefect.client import Secret
    
    environment = {}
    secret_key = Secret("SECRET_KEY")
    environment['SECRET_KEY'] = secret_key.get()
    
    with Flow(name, schedule=schedule) as flow:
      task(command='./do-the-thing', env=environment)
      flow.register(project_name=project_name)
    2. Use
    prefect.client.Secret
    to get the Secret from within the Task when it starts (similar to how an entrypoint script is often used to set environment variables in Docker environments). eg:
    # register-tasks.py
    from prefect import Flow
    from prefect.tasks.shell import ShellTask
    
    with Flow(name, schedule=schedule) as flow:
      task(command='./do-the-thing')
      flow.register(project_name=project_name)
    
    # do-the-thing
    #!/usr/bin/env python3
    from prefect.client import Secret
    
    secret_key = Secret("SECRET_KEY")
    os.environ['SECRET_KEY'] = secret_key.get()
    Is one of these a recommended over the other as the "best practice" for Prefect?
    Chris White

    Chris White

    2 years ago
    Hi Jason - yes, pulling the secret at runtime (option 2) is preferred over pulling it at build time (option 1). With option 1 you run the risk of storing the value of the secret along with the Flow, which is generally not a good idea, and ultimately defeats the purpose of using a Prefect Secret at all. Relatedly, it seems you are using
    os.environ
    to update your local environment; note that the environment that your flow runs in could be different from the environment that you built your flow in. Moreover, setting OS env vars directly in python is generally not good practice (the environment variables only take affect for subprocesses spawned from the parent process where
    os.environ
    was updated)
    j

    Jason Nochlin

    2 years ago
    How do you recommend setting the environment variables at run time? Or do you recommend not using environment variables for the value of the secret?
    Kinda related: what's the recommended way to detect that task is running from within Prefect? I can imagine having a task script that does something like:
    if running_in_prefect():
      secret_key = Secret("SECRET_KEY").get()
    else:
      secret_key = os.environ['SECRET_KEY']
    Dylan

    Dylan

    2 years ago
    @Jason Nochlin under the hood, a Secret is a prefect task whose job it is to retrieve the secret
    So it’s really just another task
    So you define the task in your flow context at build time but it’s not executed until runtime
    You can define secrets locally in a
    config.toml
    and your Prefect Secret is smart enough to check their first!
    [context.secrets]
    MY_SECET = "a very secret thing"