Thread
#prefect-community
    p

    Preston Marshall

    2 years ago
    It appears that because the parameter is never used in a task, it doesn't ever expect it
    Chris White

    Chris White

    2 years ago
    Yes, just creating a parameter is not sufficient to be added to a flow - it needs to be added explicitly or used in a downstream task
    p

    Preston Marshall

    2 years ago
    How can I make this work? Secrets are tasks too
    Currently I am reading (or trying to) directly in the run method of my secret. Would doing it in init work?
    Jeremiah

    Jeremiah

    2 years ago
    Prefect can only pass task results to the inputs of other task’s
    run
    functions, not their
    ___init___
    methods. I’m assuming
    GoogleSecretManagerSecret
    is a
    Secret
    class, so
    password_secret_path
    is being passed to
    ___init___
    , not
    run
    , and therefore not being recognized by Prefect. If you overrode your secret class to accept name as a
    run
    kwarg, then your approach of passing a dynamic name via parameter would work.
    (I’d accept a PR to implement that in base Prefect, too —
    name=None
    as an optional kwarg to
    Secret.run()
    that overwrites
    self.name
    if provided)
    p

    Preston Marshall

    2 years ago
    Thanks I’ll give that a shot
    So, I added a secret_path kwarg to my GoogleSecretManagerSecret class's run method, and it didn't work. Maybe I'm not understanding something?
    does it have to be name? because that's all secrets are wired up to do?
    Jeremiah

    Jeremiah

    2 years ago
    I’m not sure how your
    GoogleSecretManagerSecret
    is constructed so it’s hard to say, but you’ll need to provide your dynamic name to the right place. For example in the base class, this would look like this:
    def run(self, name=None):
            return _Secret(name or self.name).get()
    note
    name=None
    doesn’t currently exist in the base class
    p

    Preston Marshall

    2 years ago
    so I basically copied and pasted the EnvVar one and made the run() method retrieve it using a python api.
    so it does have a name, and I swapped secret_path for the env_var
    it's not clear to me what I'm overriding or why
    Jeremiah

    Jeremiah

    2 years ago
    I haven’t tested this code, but this is the same idea for the EnvVarSecret:
    p

    Preston Marshall

    2 years ago
    def run(self, secret_path=None):
            """
            Returns the value of a secret after applying an optional `cast` function.
    
            Returns:
                - Any: the (optionally type-cast) value of the secret
            """
            if secret_path is not None:
                self.secret_path = secret_path
    that's basically what I have, and the rest of the code uses self.secret_path.
    maybe I'm not using it correctly:
    password_secret_path = Parameter("password_secret_path")
        password = GoogleSecretManagerSecret(secret_path=password_secret_path)
    I still get the unexpected parameters error
    Jeremiah

    Jeremiah

    2 years ago
    Ah ok, I think you’re still passing your parameter to the GoogleSecretManagerSecret’s initialization
    try:
    # first initialize the secret task class
    GSMS = GoogleSecretManagerSecert()
    # now call it with the argument
    google_secret = GSMS(secret_path=password_secret_path)
    This is a gotcha of using task classes (or secret classes) - they must be instantiated prior to being called. The
    @task
    decorator lets you skip this step.
    p

    Preston Marshall

    2 years ago
    trying...
    that worked! very cool
    is there some way I could workaround having to do that in the flow?
    it looks odd to the naive viewer
    Jeremiah

    Jeremiah

    2 years ago
    It’s tough (and I agree, odd looking). One thing we’ve gotten in the habit of doing is instantiating classes first, before declaring the flow, then using them inside the flow. It feels a little more pythonic. For example:
    # set up stuff
    password_secret_path = Paramter('password_secret_path')
    sftp_connection_info = Parameter("sftp_connection_info")
    google_secret = GoogleSecretManagerSecret()
    # ... other stuf
    
    # build flow
    with Flow("mirror_sftp_to_gcs") as flow:
        password = google_secret(secret_path=password_secret_path)
        files = ...
    Basically do your declarations first, then your functional-API calls inside the flow context
    To be clear, it’s the same code as before, just organized a little differently
    We have been looking for a cleaner first-class solution to the instantiate-then-call pattern but so far haven’t stumbled on it quite yet
    p

    Preston Marshall

    2 years ago
    gotcha, makes sense
    I should just think of the constructed classes as partially applied functions
    Jeremiah

    Jeremiah

    2 years ago
    :yes: