Jason Nochlin

08/27/2020, 5:42 PM
Quick question on best practices: I'm trying to use "Secrets" with a ShellTask and see two possibilities: 1. Get the Secret when I register the flow, eg:
from prefect import Flow
from import ShellTask
from prefect.client import Secret

environment = {}
secret_key = Secret("SECRET_KEY")
environment['SECRET_KEY'] = secret_key.get()

with Flow(name, schedule=schedule) as flow:
  task(command='./do-the-thing', env=environment)
2. Use
to get the Secret from within the Task when it starts (similar to how an entrypoint script is often used to set environment variables in Docker environments). eg:
from prefect import Flow
from import ShellTask

with Flow(name, schedule=schedule) as flow:

# do-the-thing
#!/usr/bin/env python3
from prefect.client import Secret

secret_key = Secret("SECRET_KEY")
os.environ['SECRET_KEY'] = secret_key.get()
Is one of these a recommended over the other as the "best practice" for Prefect?

Chris White

08/27/2020, 5:49 PM
Hi Jason - yes, pulling the secret at runtime (option 2) is preferred over pulling it at build time (option 1). With option 1 you run the risk of storing the value of the secret along with the Flow, which is generally not a good idea, and ultimately defeats the purpose of using a Prefect Secret at all. Relatedly, it seems you are using
to update your local environment; note that the environment that your flow runs in could be different from the environment that you built your flow in. Moreover, setting OS env vars directly in python is generally not good practice (the environment variables only take affect for subprocesses spawned from the parent process where
was updated)
:thank-you: 1
👍 1

Jason Nochlin

08/27/2020, 6:17 PM
How do you recommend setting the environment variables at run time? Or do you recommend not using environment variables for the value of the secret?
Kinda related: what's the recommended way to detect that task is running from within Prefect? I can imagine having a task script that does something like:
if running_in_prefect():
  secret_key = Secret("SECRET_KEY").get()
  secret_key = os.environ['SECRET_KEY']


08/27/2020, 6:19 PM
@Jason Nochlin under the hood, a Secret is a prefect task whose job it is to retrieve the secret
So it’s really just another task
So you define the task in your flow context at build time but it’s not executed until runtime
You can define secrets locally in a
and your Prefect Secret is smart enough to check their first!
MY_SECET = "a very secret thing"