is there a way to hide parameters? Let's say a flow requires a secret to work, and that secret is given as a param. Can I hide that from the logs and UI?
a
Anna Geller
04/19/2022, 10:35 AM
Don't pass secrets as parameters 😄
There are several mechanisms for storing secrets:
• using Prefect secrets
• using external secrets manager
• leveraging environment variables
All of the above are better than using Secrets in parameters.
And to answer your question directly: there is no way of hiding that because there is no need for it - parameter values are just default configuration that can be optionally overridden at runtime. Here is a deep dive
d
Daniel Nilsen
04/19/2022, 10:37 AM
the secrets are dynamic and changes based on the flow run 🤔
a
Anna Geller
04/19/2022, 10:37 AM
if you have some sensitive value but not a secret, perhaps the KV Store is an option worth considering?
Anna Geller
04/19/2022, 10:41 AM
are you dynamically changing say DB password? I can't think of a single use case when secrets would change dynamically with each flow run - can you explain your use case more?
especially then you should consider a secrets manager to update such frequently rotating secrets
d
Daniel Nilsen
04/19/2022, 10:44 AM
are you dynamically changing say DB password
lets say I did this. how would I hide the password from the logs?
a
Anna Geller
04/19/2022, 10:47 AM
for log specifically, you could add a log filter - here is an example I got from Kevin:
Copy code
from prefect import task, Flow, Parameter
import time
import logging
import prefect
# Creating a filter
class SecureFilter(logging.Filter):
def filter(self, rec):
if 'secs' in rec.msg:
return 0
return 1
@task
def abc(x):
time.sleep(x)
return x
def get_logger():
logger = logging.getLogger("prefect.TaskRunner")
logger.addFilter(SecureFilter())
return logger
with Flow("timer flow") as flow:
logger = get_logger()
secs= Parameter("secs", 1)
abc.map([secs]*5)
flow.run()
but again, I don't think that this is the right approach (just to be transparent)
d
Daniel Nilsen
04/19/2022, 10:48 AM
ah, nice! Then it would be hidden from the UI as well?
a
Anna Geller
04/19/2022, 11:09 AM
log filters filter out logs so that those are not collected and not sent to Prefect Cloud backend
k
Kevin Kho
04/19/2022, 1:57 PM
You can dynamically change the Secret name you are passing right?
Copy code
@task
def get_secret_name(env):
return env+"password"
with Flow(...) as flow:
env = Parameter("env", default="staging")
secret_name = get_secret_name(env)
pw = PrefectSecret()(secret_name)
You just need to pass the secret name to the run method instead of the init
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.