Preston Marshall
02/08/2020, 4:46 PMPreston Marshall
02/08/2020, 4:46 PMPreston Marshall
02/08/2020, 7:27 PMPreston Marshall
02/08/2020, 7:35 PMwith Flow("mirror_sftp_to_gcs") as flow:
password_secret_path = Parameter("password_secret_path")
sftp_connection_info = Parameter("sftp_connection_info")
password = GoogleSecretManagerSecret(password_secret_path) # TODO: Cast function to convert to namedtuple
host_path = Parameter('host_path')
files = sftp_list_files(sftp_connection_info, password, host_path)
downloaded_files = sftp_to_gcs.map(
file_path=files,
connection_info=unmapped(sftp_connection_info),
password=unmapped(password),
bucket=unmapped("snip")
)
I get an error: ValueError: Flow.run received the following unexpected parameters: password_secret_path
Very odd, the other parameters work, I am specifying them in a parameters kwarg in flow.run() in another file.Preston Marshall
02/08/2020, 7:42 PMPreston Marshall
02/09/2020, 3:39 PMNate Atkins
02/09/2020, 3:43 PMPreston Marshall
02/09/2020, 3:46 PMJeremiah
02/09/2020, 3:46 PMPreston Marshall
02/09/2020, 3:51 PMJeremiah
02/09/2020, 3:55 PMPreston Marshall
02/09/2020, 4:48 PMitay livni
02/09/2020, 7:30 PMPreston Marshall
02/09/2020, 8:28 PMSecret
to the slack notifier or look it up by name? Looking at the code it seems to read from the context, but it's not clear if I can actually set secrets in the context other than via the config filePreston Marshall
02/09/2020, 8:30 PMPreston Marshall
02/09/2020, 8:31 PMChris O'Brien
02/09/2020, 9:47 PMThomas La Piana
02/10/2020, 9:56 AMMark Williams
02/10/2020, 5:38 PMdhume
02/10/2020, 6:59 PMValueError: Local Secret "SLACK_WEBHOOK_URL" was not found.
I tried the suggestions mentioned here https://github.com/PrefectHQ/prefect/blob/b4c08b15b386107e2c1afb7fed35dca15b683cdd/src/prefect/client/secrets.py but wasn’t getting anything to work locally. Threading what I currently haveJohn Ramirez
02/11/2020, 4:01 PMNate Atkins
02/11/2020, 4:28 PMtrapped
02/11/2020, 4:46 PMAdam Roderick
02/12/2020, 5:22 PMAlexander Verbitsky
02/12/2020, 7:09 PMAlexander Verbitsky
02/12/2020, 7:12 PMresult_handler
for task, but it doesn't check that data already exists in a repeatable runJan Harkema
02/14/2020, 9:58 AMCab Maddux
02/14/2020, 1:42 PMNate Atkins
02/14/2020, 8:27 PMNikita Vostretsov
02/17/2020, 6:48 AMt1
-t3
) or lower branch (t1
, t4
and t5
). Is it possible to do without rewriting code from
with Flow('flow') as flow:
r1 = t1()
r2 = t2(r1)
r3 = t3(r2)
r4 = t1(r1)
r5 = t5(r4)
into
def upper_branch():
r1 = t1()
r2 = t2(r1)
r3 = t3(r2)
def lower_branch():
r1 = t1()
r4 = t4(r1)
r5 = t5(r2)
with Flow('flow') as flow:
if do_upper or do_both:
upper_branch()
if do_lower or do_both:
lower_branch()
Nikita Vostretsov
02/17/2020, 6:48 AMt1
-t3
) or lower branch (t1
, t4
and t5
). Is it possible to do without rewriting code from
with Flow('flow') as flow:
r1 = t1()
r2 = t2(r1)
r3 = t3(r2)
r4 = t1(r1)
r5 = t5(r4)
into
def upper_branch():
r1 = t1()
r2 = t2(r1)
r3 = t3(r2)
def lower_branch():
r1 = t1()
r4 = t4(r1)
r5 = t5(r2)
with Flow('flow') as flow:
if do_upper or do_both:
upper_branch()
if do_lower or do_both:
lower_branch()
trapped
02/17/2020, 2:57 PMJeremiah
02/17/2020, 3:56 PMNikita Vostretsov
02/18/2020, 7:28 AMif
. I was thinking about something like
with Flow('flow') as flow:
r1 = t1()
r2 = t2(r1)
r3 = t3(r2)
r4 = t1(r1)
r5 = t5(r4)
targets = []
if do_upper or do_both:
targets.append(r3)
if do_upper or do_both:
targets.append(r5)
flow.run(targets=targets)
Jeremiah
02/18/2020, 4:29 PMif
is not available inside a flow since it is executed at “build time”; Prefect’s ifelse
is executed at “run time” and can therefore depend on information available when the flow runs. In your example, the flow actually runs both branches no matter what, as you have no runtime control flow implemented. The only thing changing is the input to your targets
keyword argument, which is presumably a Parameter not shown here.ifelse
is intended for; you want to make a decision at runtime about which branch to execute. If, on the other hand, you know which branch you want to run AND you don’t mind rebuilding your flow each time (which might be fine for your purposes!), then you could use a Python if
to control which tasks you add to the flow. However, if you want a single flow with multiple branches, you’ll need a Prefect ifelse
.