https://prefect.io logo
#prefect-community
Title
# prefect-community
j

Jason Nochlin

08/27/2020, 5:42 PM
Quick question on best practices: I'm trying to use "Secrets" with a ShellTask and see two possibilities: 1. Get the Secret when I register the flow, eg:
Copy code
from prefect import Flow
from prefect.tasks.shell import ShellTask
from prefect.client import Secret

environment = {}
secret_key = Secret("SECRET_KEY")
environment['SECRET_KEY'] = secret_key.get()

with Flow(name, schedule=schedule) as flow:
  task(command='./do-the-thing', env=environment)
  flow.register(project_name=project_name)
2. Use
prefect.client.Secret
to get the Secret from within the Task when it starts (similar to how an entrypoint script is often used to set environment variables in Docker environments). eg:
Copy code
# register-tasks.py
from prefect import Flow
from prefect.tasks.shell import ShellTask

with Flow(name, schedule=schedule) as flow:
  task(command='./do-the-thing')
  flow.register(project_name=project_name)

# do-the-thing
#!/usr/bin/env python3
from prefect.client import Secret

secret_key = Secret("SECRET_KEY")
os.environ['SECRET_KEY'] = secret_key.get()
Is one of these a recommended over the other as the "best practice" for Prefect?
c

Chris White

08/27/2020, 5:49 PM
Hi Jason - yes, pulling the secret at runtime (option 2) is preferred over pulling it at build time (option 1). With option 1 you run the risk of storing the value of the secret along with the Flow, which is generally not a good idea, and ultimately defeats the purpose of using a Prefect Secret at all. Relatedly, it seems you are using
os.environ
to update your local environment; note that the environment that your flow runs in could be different from the environment that you built your flow in. Moreover, setting OS env vars directly in python is generally not good practice (the environment variables only take affect for subprocesses spawned from the parent process where
os.environ
was updated)
🙏 1
👍 1
j

Jason Nochlin

08/27/2020, 6:17 PM
How do you recommend setting the environment variables at run time? Or do you recommend not using environment variables for the value of the secret?
Kinda related: what's the recommended way to detect that task is running from within Prefect? I can imagine having a task script that does something like:
Copy code
if running_in_prefect():
  secret_key = Secret("SECRET_KEY").get()
else:
  secret_key = os.environ['SECRET_KEY']
d

Dylan

08/27/2020, 6:19 PM
@Jason Nochlin under the hood, a Secret is a prefect task whose job it is to retrieve the secret
So it’s really just another task
So you define the task in your flow context at build time but it’s not executed until runtime
You can define secrets locally in a
config.toml
and your Prefect Secret is smart enough to check their first!
Copy code
[context.secrets]
MY_SECET = "a very secret thing"