https://prefect.io logo
Powered by
Title
p

Preston Marshall

02/08/2020, 7:42 PM
It appears that because the parameter is never used in a task, it doesn't ever expect it
c

Chris White

02/08/2020, 7:47 PM
Yes, just creating a parameter is not sufficient to be added to a flow - it needs to be added explicitly or used in a downstream task
p

Preston Marshall

02/08/2020, 8:01 PM
How can I make this work? Secrets are tasks too
Currently I am reading (or trying to) directly in the run method of my secret. Would doing it in init work?
j

Jeremiah

02/08/2020, 9:12 PM
Prefect can only pass task results to the inputs of other task’s 
run
functions, not their 
___init___
methods. I’m assuming 
GoogleSecretManagerSecret
is a 
Secret
class, so 
password_secret_path
is being passed to 
___init___
, not 
run
, and therefore not being recognized by Prefect. If you overrode your secret class to accept name as a 
run
kwarg, then your approach of passing a dynamic name via parameter would work.
(I’d accept a PR to implement that in base Prefect, too — 
name=None
as an optional kwarg to 
Secret.run()
that overwrites 
self.name
if provided)
p

Preston Marshall

02/08/2020, 9:26 PM
Thanks I’ll give that a shot
So, I added a secret_path kwarg to my GoogleSecretManagerSecret class's run method, and it didn't work. Maybe I'm not understanding something?
does it have to be name? because that's all secrets are wired up to do?
j

Jeremiah

02/08/2020, 9:46 PM
I’m not sure how your 
GoogleSecretManagerSecret
is constructed so it’s hard to say, but you’ll need to provide your dynamic name to the right place. For example in the base class, this would look like this:
def run(self, name=None):
        return _Secret(name or self.name).get()
note 
name=None
doesn’t currently exist in the base class
p

Preston Marshall

02/08/2020, 9:47 PM
so I basically copied and pasted the EnvVar one and made the run() method retrieve it using a python api.
so it does have a name, and I swapped secret_path for the env_var
it's not clear to me what I'm overriding or why
j

Jeremiah

02/08/2020, 9:51 PM
I haven’t tested this code, but this is the same idea for the EnvVarSecret:
p

Preston Marshall

02/08/2020, 9:52 PM
def run(self, secret_path=None):
        """
        Returns the value of a secret after applying an optional `cast` function.

        Returns:
            - Any: the (optionally type-cast) value of the secret
        """
        if secret_path is not None:
            self.secret_path = secret_path
that's basically what I have, and the rest of the code uses self.secret_path.
maybe I'm not using it correctly: 
password_secret_path = Parameter("password_secret_path")
    password = GoogleSecretManagerSecret(secret_path=password_secret_path)
I still get the unexpected parameters error
j

Jeremiah

02/08/2020, 9:53 PM
Ah ok, I think you’re still passing your parameter to the `GoogleSecretManagerSecret`’s initialization
try:
# first initialize the secret task class
GSMS = GoogleSecretManagerSecert()
# now call it with the argument
google_secret = GSMS(secret_path=password_secret_path)
This is a gotcha of using task classes (or secret classes) - they must be instantiated prior to being called. The 
@task
decorator lets you skip this step.
p

Preston Marshall

02/08/2020, 9:54 PM
trying...
that worked! very cool
is there some way I could workaround having to do that in the flow?
it looks odd to the naive viewer
j

Jeremiah

02/08/2020, 9:58 PM
It’s tough (and I agree, odd looking). One thing we’ve gotten in the habit of doing is instantiating classes first, before declaring the flow, then using them inside the flow. It feels a little more pythonic. For example:
# set up stuff
password_secret_path = Paramter('password_secret_path')
sftp_connection_info = Parameter("sftp_connection_info")
google_secret = GoogleSecretManagerSecret()
# ... other stuf

# build flow
with Flow("mirror_sftp_to_gcs") as flow:
    password = google_secret(secret_path=password_secret_path)
    files = ...
Basically do your declarations first, then your functional-API calls inside the flow context
To be clear, it’s the same code as before, just organized a little differently
We have been looking for a cleaner first-class solution to the instantiate-then-call pattern but so far haven’t stumbled on it quite yet
p

Preston Marshall

02/08/2020, 10:01 PM
gotcha, makes sense
I should just think of the constructed classes as partially applied functions
j

Jeremiah

02/08/2020, 10:10 PM
:yes:
#prefect-communityJoin Slack
Powered by