Preston Marshall
02/08/2020, 4:46 PMPreston Marshall
02/08/2020, 7:27 PMPreston Marshall
02/08/2020, 7:35 PMwith Flow("mirror_sftp_to_gcs") as flow:
password_secret_path = Parameter("password_secret_path")
sftp_connection_info = Parameter("sftp_connection_info")
password = GoogleSecretManagerSecret(password_secret_path) # TODO: Cast function to convert to namedtuple
host_path = Parameter('host_path')
files = sftp_list_files(sftp_connection_info, password, host_path)
downloaded_files = sftp_to_gcs.map(
file_path=files,
connection_info=unmapped(sftp_connection_info),
password=unmapped(password),
bucket=unmapped("snip")
)
I get an error: ValueError: Flow.run received the following unexpected parameters: password_secret_path
Very odd, the other parameters work, I am specifying them in a parameters kwarg in flow.run() in another file.Preston Marshall
02/08/2020, 7:42 PMPreston Marshall
02/09/2020, 3:39 PMNate Atkins
02/09/2020, 3:43 PMPreston Marshall
02/09/2020, 3:46 PMJeremiah
Preston Marshall
02/09/2020, 3:51 PMJeremiah
Preston Marshall
02/09/2020, 4:48 PMitay livni
02/09/2020, 7:30 PMPreston Marshall
02/09/2020, 8:28 PMSecret
to the slack notifier or look it up by name? Looking at the code it seems to read from the context, but it's not clear if I can actually set secrets in the context other than via the config filePreston Marshall
02/09/2020, 8:30 PMPreston Marshall
02/09/2020, 8:31 PMChris O'Brien
02/09/2020, 9:47 PMThomas La Piana
02/10/2020, 9:56 AMMark Williams
02/10/2020, 5:38 PMdhume
02/10/2020, 6:59 PMValueError: Local Secret "SLACK_WEBHOOK_URL" was not found.
I tried the suggestions mentioned here https://github.com/PrefectHQ/prefect/blob/b4c08b15b386107e2c1afb7fed35dca15b683cdd/src/prefect/client/secrets.py but wasnât getting anything to work locally. Threading what I currently haveJohn Ramirez
02/11/2020, 4:01 PMNate Atkins
02/11/2020, 4:28 PMtrapped
02/11/2020, 4:46 PMAdam Roderick
02/12/2020, 5:22 PMAlexander Verbitsky
02/12/2020, 7:09 PMAlexander Verbitsky
02/12/2020, 7:12 PMresult_handler
for task, but it doesn't check that data already exists in a repeatable runJan Harkema
02/14/2020, 9:58 AMCab Maddux
02/14/2020, 1:42 PMNate Atkins
02/14/2020, 8:27 PMNikita Vostretsov
02/17/2020, 6:48 AMt1
-t3
) or lower branch (t1
, t4
and t5
). Is it possible to do without rewriting code from
with Flow('flow') as flow:
r1 = t1()
r2 = t2(r1)
r3 = t3(r2)
r4 = t1(r1)
r5 = t5(r4)
into
def upper_branch():
r1 = t1()
r2 = t2(r1)
r3 = t3(r2)
def lower_branch():
r1 = t1()
r4 = t4(r1)
r5 = t5(r2)
with Flow('flow') as flow:
if do_upper or do_both:
upper_branch()
if do_lower or do_both:
lower_branch()
John Ramirez
02/17/2020, 5:12 PM