https://prefect.io logo
Channels
Powered by
m

Matt

09/29/2020, 10:36 PM
Hi! I'm new to Prefect, working on my first flow, and struggling a bit with config and context in Prefect Core. I'm calling this from the command line and have the parameters going in fine and am grabbing secrets from env settings. I also understand that Prefect returns these as Tasks and not strings. Are both of these (Parameters and EnvVarSecrets) only available to the Flow? I don't seem to be able to access them from the Tasks themselves and it doesn't seem like they get automatically added to prefect.context or prefect.config. Is the only way to do this to pass them to the Task (socrata.download_dataset in this case) as parameters? I've tried adding the Secrets to both context and config programmatically but they don't seem to appear when I do this although the docs suggest this should work. My first attempt at a flow had parameters getting passed every which way and I'm trying to clean this up. Would love some guidance from the community. Thanks!
Untitled
n

nicholas

09/29/2020, 11:51 PM
Hi @Matt - you can access any 
Secret
task from within another task by calling its 
.get
method, like such:
Copy code
token = Secret("SOME_SECRET").get()
so a more full example might be:
Copy code
from prefect import Flow, task
from prefect.client import Secret

@task
def my_task():
   my_secret = Secret("MY_SECRET").get()
   # do something with secret

with Flow("My Flow") as flow:
  my_task()
m

Matt

09/30/2020, 12:34 AM
hi nicholas, that looks like a good solution, but does it work if I'm running my flow from the command line? i'm doing a 'python myflow.py' approach. i'm not sure if the client only gets created if you do a prefect run flow or run it from a server. at least it just tried this approach and it couldn't find my secret.
maybe i'm missing something else?
c

Chris White

09/30/2020, 12:36 AM
Hi Matt - did you set 
SOCRATA_TOKEN
as an environment variable in the process that you ran 
python myflow.py
from?
@nicholas's suggestion uses a Prefect Secret specifically, which can be set via the
Copy code
PREFECT__CONTEXT__SECRETS__MY_SECRET
environment variable, depending on which pattern you prefer
m

Matt

09/30/2020, 12:58 AM
yep. calling
Copy code
token = EnvVarSecret("SOCRATA_TOKEN", raise_if_missing=True)
from my 'with flow()' works fine. i can get the secret and pass it to my tasks as a parameter. i'm hoping there's a way to access it from my task either from context, config or something else so i don't have to explicitly pass it each time as a parameter. for instance i have a bunch of tasks that run against a database and they all need a DSN
c

Chris White

09/30/2020, 1:03 AM
Gotcha, in that case you can put it directly into context via environment variable:
Copy code
PREFECT__CONTEXT__SECRETS__SOCRATA_TOKEN=value
and then pull it from 
prefect.context.secrets["SOCRATA_TOKEN"]
within your tasks, without making it an explicit task itself
🙌 1
m

Matt

09/30/2020, 3:09 AM
Perfect! This worked great. I'm just prefixing all my envs with PREFECT__CONTEXT__SECRETS__ and retrieving them in the tasks like you suggested. It seems there's not way to add secrets to context from code which probably makes sense from a security aspect as I think about it. My tasks param craziness is pretty much gone. Thanks!!
🎉 1
2 Views
Powered by