Jason Nochlin
08/27/2020, 5:42 PMfrom prefect import Flow
from prefect.tasks.shell import ShellTask
from prefect.client import Secret
environment = {}
secret_key = Secret("SECRET_KEY")
environment['SECRET_KEY'] = secret_key.get()
with Flow(name, schedule=schedule) as flow:
task(command='./do-the-thing', env=environment)
flow.register(project_name=project_name)
2. Use prefect.client.Secret
to get the Secret from within the Task when it starts (similar to how an entrypoint script is often used to set environment variables in Docker environments). eg:
# register-tasks.py
from prefect import Flow
from prefect.tasks.shell import ShellTask
with Flow(name, schedule=schedule) as flow:
task(command='./do-the-thing')
flow.register(project_name=project_name)
# do-the-thing
#!/usr/bin/env python3
from prefect.client import Secret
secret_key = Secret("SECRET_KEY")
os.environ['SECRET_KEY'] = secret_key.get()
Is one of these a recommended over the other as the "best practice" for Prefect?Chris White
08/27/2020, 5:49 PMos.environ
to update your local environment; note that the environment that your flow runs in could be different from the environment that you built your flow in. Moreover, setting OS env vars directly in python is generally not good practice (the environment variables only take affect for subprocesses spawned from the parent process where os.environ
was updated)Jason Nochlin
08/27/2020, 6:17 PMif running_in_prefect():
secret_key = Secret("SECRET_KEY").get()
else:
secret_key = os.environ['SECRET_KEY']
Dylan
08/27/2020, 6:19 PMconfig.toml
and your Prefect Secret is smart enough to check their first![context.secrets]
MY_SECET = "a very secret thing"