https://prefect.io logo
Join Slack
Powered by
It appears that because the parameter is never use...
# ask-community
p
It appears that because the parameter is never used in a task, it doesn't ever expect it
c
Yes, just creating a parameter is not sufficient to be added to a flow - it needs to be added explicitly or used in a downstream task
p
How can I make this work? Secrets are tasks too
Currently I am reading (or trying to) directly in the run method of my secret. Would doing it in init work?
j
Prefect can only pass task results to the inputs of other task’s 
run
functions, not their 
___init___
methods. I’m assuming 
GoogleSecretManagerSecret
is a 
Secret
class, so 
password_secret_path
is being passed to 
___init___
, not 
run
, and therefore not being recognized by Prefect. If you overrode your secret class to accept name as a 
run
kwarg, then your approach of passing a dynamic name via parameter would work.
(I’d accept a PR to implement that in base Prefect, too — 
name=None
as an optional kwarg to 
Secret.run()
that overwrites 
self.name
if provided)
p
Thanks I’ll give that a shot
So, I added a secret_path kwarg to my GoogleSecretManagerSecret class's run method, and it didn't work. Maybe I'm not understanding something?
does it have to be name? because that's all secrets are wired up to do?
j
I’m not sure how your 
GoogleSecretManagerSecret
is constructed so it’s hard to say, but you’ll need to provide your dynamic name to the right place. For example in the base class, this would look like this:
Copy code
def run(self, name=None):
        return _Secret(name or self.name).get()
note 
name=None
doesn’t currently exist in the base class
p
so I basically copied and pasted the EnvVar one and made the run() method retrieve it using a python api.
so it does have a name, and I swapped secret_path for the env_var
it's not clear to me what I'm overriding or why
j
I haven’t tested this code, but this is the same idea for the EnvVarSecret:
p
Copy code
def run(self, secret_path=None):
        """
        Returns the value of a secret after applying an optional `cast` function.

        Returns:
            - Any: the (optionally type-cast) value of the secret
        """
        if secret_path is not None:
            self.secret_path = secret_path
that's basically what I have, and the rest of the code uses self.secret_path.
maybe I'm not using it correctly:
Copy code
password_secret_path = Parameter("password_secret_path")
    password = GoogleSecretManagerSecret(secret_path=password_secret_path)
I still get the unexpected parameters error
j
Ah ok, I think you’re still passing your parameter to the `GoogleSecretManagerSecret`’s initialization
try:
Copy code
# first initialize the secret task class
GSMS = GoogleSecretManagerSecert()
# now call it with the argument
google_secret = GSMS(secret_path=password_secret_path)
This is a gotcha of using task classes (or secret classes) - they must be instantiated prior to being called. The 
@task
decorator lets you skip this step.
p
trying...
that worked! very cool
is there some way I could workaround having to do that in the flow?
it looks odd to the naive viewer
j
It’s tough (and I agree, odd looking). One thing we’ve gotten in the habit of doing is instantiating classes first, before declaring the flow, then using them inside the flow. It feels a little more pythonic. For example:
Copy code
# set up stuff
password_secret_path = Paramter('password_secret_path')
sftp_connection_info = Parameter("sftp_connection_info")
google_secret = GoogleSecretManagerSecret()
# ... other stuf

# build flow
with Flow("mirror_sftp_to_gcs") as flow:
    password = google_secret(secret_path=password_secret_path)
    files = ...
Basically do your declarations first, then your functional-API calls inside the flow context
To be clear, it’s the same code as before, just organized a little differently
We have been looking for a cleaner first-class solution to the instantiate-then-call pattern but so far haven’t stumbled on it quite yet
p
gotcha, makes sense
I should just think of the constructed classes as partially applied functions
j
👌
3 Views
Open in Slack
PreviousNext
Powered by