https://prefect.io
Join Slack
Hi, anyone has a link to documentation on best-practices regarding passing secrets as Task Input in ...
# ask-community
j
Hi, anyone has a link to documentation on best-practices regarding passing secrets as Task Input in Prefect 2 ? I would ideally like that my secrets never get sent in clear text to Prefect Cloud also.
n
Hi @Jean-Michel Provencher One way to do it would be to pass the name of the secret you need to the task (or flow) that needs it and then just call 
Secret.load(my_secret_name)
wherever you need to actually use it
j
yeah, but I'm already using a from prefect_aws.secrets_manager import read_secret task so I was wondering how to use that built-in task to actually pass the secret to another task.
👀 1
n
one way (which is not the slickest, but should work) is to just wrap the output of 
read_secret
with pydantic's 
SecretStr
and then get the secret value inside the task To help me ask a better question to our UI folks, where are you concerned with seeing plain text secrets in the UI?
j
I'm not concerned necessarly, it's just for compliance reason I'd prefer my secrets to never be sent to prefect cloud. If I use pydantic will they be sent in clear to prefect cloud ?
n
gotcha - I was just asking in case you had noticed them in the UI somewhere (they should not be exposed in the UI in the first place) but to be extra sure, you could wrap your input in 
SecretStr
, because then as far as Prefect Cloud is concerned, it would only know that you passed a 
SecretStr
with some 
name
(or whatever the input arg is called) as a Task Input
j
Interesting.
m
It would be good to have additional guidance here, we only just ran into this as well and would like to have control over preventing sensitive information being stored in prefect. Is there a document somewhere describing exactly what data is stored in Prefect? For us, being able to disable task inputs being logged, or mask based on field names, would be really great. (CC @Emil Christensen)
j
I'm also wondering about the underlying mecanisms of task input and be sure that if I pass a SecretStr as a task input, it would would not send the actual task input value to prefect cloud, and instead only use my storage for that. I found this resource, but it would be nice to have some. kind of "official" documentation on the subject.
z
Hi! At this time, values passed to tasks are never sent to the Prefect API.
Only flow parameters are stored in the Prefect API.
Similarly, we never store return values in the API (except for null/bools). See https://docs.prefect.io/concepts/results/#storage-of-results-in-prefect for more details on that.
The “Task Inputs” field is used to track relationships between task runs right now. For example, if you pass the result of one task run into another task run we will track the relationship. However, the value itself is only tracked locally. The API has scaffolding to track other types of inputs there, i.e. actual values. However, we have not built any support for that into the client and if/when we do so it is very likely to be opt-in.
🙏 1
m
ok, thanks for the clarification, but as a note, it does look like task inputs are being tracked (hence our confusion). e.g. below, the 
input_substance
is something we definitely do not want to be tracked, and this ui suggests maybe it could be.
j
@Ton Steijvers
n
Hi @Mike O'Connor - we agree, and we're currently working on a better way to represent the purpose of the task inputs page you can track here
🙌 1
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by