Preston Marshall
02/08/2020, 7:42 PMChris White
02/08/2020, 7:47 PMPreston Marshall
02/08/2020, 8:01 PMJeremiah
02/08/2020, 9:12 PMrun
functions, not their ___init___
methods. I’m assuming GoogleSecretManagerSecret
is a Secret
class, so password_secret_path
is being passed to ___init___
, not run
, and therefore not being recognized by Prefect. If you overrode your secret class to accept name as a run
kwarg, then your approach of passing a dynamic name via parameter would work.name=None
as an optional kwarg to Secret.run()
that overwrites self.name
if provided)Preston Marshall
02/08/2020, 9:26 PMJeremiah
02/08/2020, 9:46 PMGoogleSecretManagerSecret
is constructed so it’s hard to say, but you’ll need to provide your dynamic name to the right place. For example in the base class, this would look like this:def run(self, name=None):
return _Secret(name or self.name).get()
name=None
doesn’t currently exist in the base classPreston Marshall
02/08/2020, 9:47 PMJeremiah
02/08/2020, 9:51 PMPreston Marshall
02/08/2020, 9:52 PMdef run(self, secret_path=None):
"""
Returns the value of a secret after applying an optional `cast` function.
Returns:
- Any: the (optionally type-cast) value of the secret
"""
if secret_path is not None:
self.secret_path = secret_path
password_secret_path = Parameter("password_secret_path")
password = GoogleSecretManagerSecret(secret_path=password_secret_path)
Jeremiah
02/08/2020, 9:53 PM# first initialize the secret task class
GSMS = GoogleSecretManagerSecert()
# now call it with the argument
google_secret = GSMS(secret_path=password_secret_path)
@task
decorator lets you skip this step.Preston Marshall
02/08/2020, 9:54 PMJeremiah
02/08/2020, 9:58 PM# set up stuff
password_secret_path = Paramter('password_secret_path')
sftp_connection_info = Parameter("sftp_connection_info")
google_secret = GoogleSecretManagerSecret()
# ... other stuf
# build flow
with Flow("mirror_sftp_to_gcs") as flow:
password = google_secret(secret_path=password_secret_path)
files = ...
Preston Marshall
02/08/2020, 10:01 PMJeremiah
02/08/2020, 10:10 PM